Multimodal Interaction Research Articles

Understanding Facilitator Interventions in the Swedish Service “Taltjänst”

The Swedish Communication Facilitation Service “Taltjänst” provides support to people with communication disorders, commonly caused by neurological disabilities, and their conversation partners. This study aims to investigate interaction within ”Taltjänst”, focusing on facilitator interventions by exploring; 1) When do facilitators intervene? 2) When do the primary conversation participants manage the conversation without facilitator support? Multimodal interaction/ Conversation analytical methods were used to analyse two phone calls involving a facilitator and clients with communication disorders. One client had congenital dysarthria, while the other had acquired aphasia, allowing for comparison of the interactional challenges faced by the facilitator when addressing different disorders. The analyses revealed that; 1) facilitators are commonly invited to intervene by their client’s eye gaze, and 2) there is a routine of intervening after every non-interrupted turn of the client with dysarthria, while the intervention pattern for the client with aphasia is more varied. These findings provide insights into the work of communication facilitators, and anyone supporting phone calls involving participants with a communication disorders. Furthermore, an increased understanding of interactional practices can contribute to the development of this essential, yet underexplored service.

Opening interspecies encounters – Greetings between humans and nonhuman animals

This paper investigates how interspecies greeting routines between humans and horses/dogs are organized. The aim is to understand how humans and animals jointly organize ‘becoming socially co-present’ (Pillet-Shore, 2008). The data comprises 12 h of video recordings with 30 humans, 40 horses and 7 dogs participating in every-day activities such as walks, dinner parties and grooming sessions. Multimodal interaction analysis (e.g., Mondada, 2019) is used to investigate how humans and horses/dogs initiate and respond to greetings, and image based transcriptions of human and animal actions are used to avoid species hierarchy. The analyses reveal that humans and animals make a distinction between being physically co-present and socially co-present in interspecies interaction, and that all involved species take initiatives to greet and respond to such initiatives. The findings show both similarities and differences to human–human greetings, which is are addressed in a concluding discussion. By including animals’ systematic communicative actions in the investigation of social life and co-operative action we gain knowledge on how understanding both within and across species boundaries is achieved, which may have implications for how humanity views other living species on the planet.

A Systematic Process to Engineer Dependable Integration of Frame-based Input Devices in a Multimodal Input Chain: Application to Rehabilitation in Healthcare

Designing new input devices and associated interaction techniques is a key contribution in order to increase the bandwidth between users and interactive applications. In the field of Human-Computer Interaction, research and development services in industry and research laboratories in universities have been, since the invention of the mouse and the graphical user interface, proposing multiple contributions including the integration of multiple input devices in multimodal interaction technique. Those contributions, most of the time, are presented as prototypes or demonstrators providing evidence of the bandwidth increase through user studies. Such contributions however, do not provide any support to software developers for integrating them in real-life systems. When done, this integration is performed in a craft manner, outside required software engineering good practice. This paper proposes a systematic process to integrate novel input devices and associated interaction techniques to better support users' work. It exploits most recent interactive systems architectural model and formal model-based approaches for interactive systems supporting verification and validation when required. The paper focusses on Frame-based input devices which support gesture-based interactions and movements recognition but also addresses their multimodal use. This engineering approach is demonstrated on an interactive application in the area of rehabilitation in healthcare where dependability of interactions and applications is as critical as their usability.

ViRgilites: Multilevel Feedforward for Multimodal Interaction in VR

Navigating the interaction landscape of Virtual Reality (VR) and Augmented Reality (AR) presents significant complexities due to the plethora of available input hardware and interaction modalities, compounded by spatially diverse visual interfaces. Such complexities elevate the likelihood of user errors, necessitating frequent backtracking. To address this, we introduce ViRgilites, a virtual guidance framework that delivers multi-level feedforward information covering the available interaction techniques as well as the future possibilities to interact with virtual objects, anticipating the interaction effects and how they fit with the overall user's goal. ViRgilites is engineered to facilitate task execution, empowering users to make informed decisions about action methodologies and alternative courses of action. This paper presents the architecture and functionality of ViRgilites and demonstrates its efficacy through evaluation with a formative user study.

How assisted eating becomes a caring practice in institutional settings: Embodied gestures and stages of assisted eating

Assisted eating is a basic caring practice and the means through which many individuals receive adequate nutrition. Research in this area has noted the challenges of helping others to eat while upholding their independence, though has yet to explicate how this caring practice is achieved in detail and across the lifespan. This paper provides an empirical analysis of assisted eating episodes in two different institutions, detailing the processes through which eating is collaboratively achieved between two persons. Data are video-recorded episodes of infants during preschool lunches and care home meals for adults with dementia, both located in Sweden. Using EMCA's multimodal interaction analysis, three core stages of assisted eating and their underpinning embodied practices were identified: (1) establishing joint attention, (2) offering the food, and (3) transferring food into the mouth. The first stage is particularly crucial in establishing the activity as a collaborative process. The analysis details the interactional practices through which assisted eating becomes a joint accomplishment using a range of multimodal features such as eye gaze, hand gestures, and vocalisations. The paper thus demonstrates how assisted eating becomes a caring practice through the active participation of both caregiver and cared-for person, according to their needs. The analysis has implications not only for professional caring work in institutional settings but also for the detailed analysis of eating as an embodied activity.

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language understanding (NLU), dialogue management, and natural language generation (NLG). We designed and implemented an HRI system that leverages models such as BERT for language understanding and GPT-3 for generating contextually appropriate responses. Our methodology involves integrating these NLP models with a robotics platform, ensuring real-time interaction capabilities while maintaining a high level of accuracy and context awareness. The system was evaluated through a series of user studies, measuring performance metrics such as accuracy, latency, and user satisfaction. Results indicate that our NLP-enhanced HRI system significantly improves the quality of interactions, demonstrating superior understanding and responsiveness compared to traditional systems. This paper discusses the implementation challenges, including computational constraints and ambiguity resolution, and provides insights into user feedback and system performance. Future work will focus on enhancing context management, exploring multimodal interaction, and addressing ethical considerations in deploying advanced HRI systems. Our findings underscore the potential of NLP to transform human-robot communication, paving the way for more intuitive and effective robotic assistants in various domains. Keywords: Human-Robot Interaction (HRI), Natural Language Processing (NLP), Conversational AI, Speech Recognition, Natural Language Understanding (NLU), Natural Language Generation (NLG), Multimodal Interaction, Dialogue Systems, Context Awareness, Emotion Recognition, Machine Learning in HRI, Personalized Interaction, User Experience (UX) in HRI, Human-Centered Design, Collaborative Robots (Cobots)

Virtual/augmented reality-based human–machine interface and interaction modes in airport control towers

The concept of an innovative human–machine interface and interaction modes based on virtual and augmented reality technologies for airport control towers has been developed with the aim of increasing the human performances and situational awareness of air traffic control operators. By presenting digital information through see-through head-mounted displays superimposed over the out-of-the-tower view, the proposed interface should stimulate controllers to operate in a head-up position and, therefore, reduce the number of switches between a head-up and a head-down position even in low visibility conditions. This paper introduces the developed interface and describes the exercises conducted to validate the technical solutions developed, focusing on the simulation platform and exploited technologies, to demonstrate how virtual and augmented reality, along with additional features such as adaptive human–machine interface, multimodal interaction and attention guidance, enable a more natural and effective interaction in the control tower. The results of the human-in-the-loop real-time validation exercises show that the prototype concept is feasible from both an operational and technical perspective, the solution proves to support the air traffic controllers in working in a head-up position more than head-down even with low-visibility operational scenarios, and to lower the time to react in critical or alerting situations with a positive impact on the human performances of the user. While showcasing promising results, this study also identifies certain limitations and opportunities for refinement, aimed at further optimising the efficacy and usability of the proposed interface.

Digital education through guided pretend play

BackgroundThe importance of play for children and its potential for learning are widely recognized. However, it is contested whether teachers should guide play, particularly pretend play, and how guided pretend play supports children's agency and learning. In this study, pretend play is employed in (unplugged) digital education and the teacher's guidance and children's involvement are examined to answer the following questions: what roles do teachers take on during guided pretend play and how do they enable children's learning about digitalization and digital transformation? AimsThis qualitative study examines how teachers guide pretend play in ways that support learning, focusing on the topic of digitalization and digital transformation. SampleFifteen teachers took part in the study and implemented the pretend play suggestions in their kindergartens with children aged from four to six years. MethodThe pretend play was filmed. The video data were structured and sequences of teachers joining in the pretend play and guiding from within were selected. These sequences were analysed in-depth using multimodal interaction analysis. ResultsThe results indicate that teachers guide pretend play from within by taking on different roles (play leader or co-player). Teachers model within pretend play, for example, by assigning tasks or thinking aloud, and provide scaffolding, encouraging children's learning about digitalization and digital transformation. ConclusionThrough co-playing and leading, teachers support a prolonged and in-depth joint focus, as well as the agency of the children. Such guided play is termed “sustained shared playing” and has great potential for learning.

Multi-modal interaction with token division strategy for RGB-T tracking

RGB-T tracking takes visible and infrared images as inputs, which is an extended application of multi-modal fusion in the field of visual object tracking. The complementarity between visible and infrared modalities can enhance the robustness of tracker in complex scenes. Cross-modal interaction can facilitate the fusion and synergy of different modalities, but most previous methods lack clear target information in multi-modal fusion, leading to some undesired cross-relation in interaction. To reduce these undesired cross-relations, we propose a Multi-modal Interaction scheme Guided by Token Division strategy (MIGTD). This scheme divides the input multi-modal tokens into several categories and restricts the interaction between tokens by setting different rules. The above operation is implemented in parallel through an attention masking strategy. To accurately classify search tokens, an instance segmentation task with box-supervised loss is employed. We conduct extensive experiments on three popular benchmark datasets, RGBT234, LasHeR and VTUAV. The experimental results indicate that the tracker proposed in this article reach the world’s advanced level in performance.

Word frequency and cognitive effort in turns-at-talk: turn structure affects processing load in natural conversation.

Frequency distributions are known to widely affect psycholinguistic processes. The effects of word frequency in turns-at-talk, the nucleus of social action in conversation, have, by contrast, been largely neglected. This study probes into this gap by applying corpus-linguistic methods on the conversational component of the British National Corpus (BNC) and the Freiburg Multimodal Interaction Corpus (FreMIC). The latter includes continuous pupil size measures of participants of the recorded conversations, allowing for a systematic investigation of patterns in the contained speech and language on the one hand and their relation to concurrent processing costs they may incur in speakers and recipients on the other hand. We test a first hypothesis in this vein, analyzing whether word frequency distributions within turns-at-talk are correlated with interlocutors' processing effort during the production and reception of these turns. Turns are found to generally show a regular distribution pattern of word frequency, with highly frequent words in turn-initial positions, mid-range frequency words in turn-medial positions, and low-frequency words in turn-final positions. Speakers' pupil size is found to tend to increase during the course of a turn at talk, reaching a climax toward the turn end. Notably, the observed decrease in word frequency within turns is inversely correlated with the observed increase in pupil size in speakers, but not in recipients, with steeper decreases in word frequency going along with steeper increases in pupil size in speakers. We discuss the implications of these findings for theories of speech processing, turn structure, and information packaging. Crucially, we propose that the intensification of processing effort in speakers during a turn at talk is owed to an informational climax, which entails a progression from high-frequency, low-information words through intermediate levels to low-frequency, high-information words. At least in English conversation, interlocutors seem to make use of this pattern as one way to achieve efficiency in conversational interaction, creating a regularly recurring distribution of processing load across speaking turns, which aids smooth turn transitions, content prediction, and effective information transfer.

DEVELOPING MULTIMODAL LISTENING SKILLS IN ENGLISH FOR SPECIFIC (OR SPECIAL) PURPOSES: A PEDAGOGICAL FRAMEWORK

In the context of English for Specific Purposes (ESP), the development of multimodal listening skills is increasingly critical due to the prevalence of multimedia communication in professional settings. This article proposes a comprehensive pedagogical framework for enhancing multimodal listening abilities in ESP learners. The framework is built on key components including needs analysis, the use of authentic materials, an integrated skills approach, leveraging technological tools, scaffolded learning, active listening strategies, and feedback mechanisms. By incorporating these elements, educators can effectively prepare learners for the complex multimodal interactions they will encounter in their specific fields. This approach not only improves listening comprehension but also facilitates a deeper engagement with language in context, ensuring that learners are equipped with the necessary skills to succeed in their professional endeavors.

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Desktop Voice Assistant

Abstract: Desktop voice assistants have become a popular interface for human-computer interaction because they allow users to speak with their computers in natural language. However, modern desktop voice assistants sometimes struggle to understand and respond to complex queries and context. Privacy concerns concerning the collection and use of user data also remain significant barriers. In order to enhance desktop voice assistants, we present a novel personalization and multimodal interaction technique in this research. By combining text, voice, and visual inputs, our approach improves the accuracy and relevance of the responses. By combining several modalities, our system is able to better understand context and user intent, leading to more meaningful interactions. Furthermore, we propose a user-centric personalization strategy whereby the voice assistant progressively learns each user's preferences and usage patterns. Because it depends less on centralized data collection and processing, this customized approach improves user experience while simultaneously addressing privacy concerns. Through a series of experiments and user surveys, we show how effective our approach is in improving the overall performance and user experience of desktop voice assistants. Our results show that the combination of multimodal interaction with personalization produces more accurate and contextually relevant responses, improving the user experience. All things considered, our research advances desktop voice assistant technology by addressing significant interface and personalization issues. Our approach has the potential to significantly improve desktop voice assistant usability and user acceptance, which could result in considerably more intuitive and natural human-computer interaction.

Alzheimer’s disease diagnosis from multi-modal data via feature inductive learning and dual multilevel graph neural network

Multi-modal data can provide complementary information of Alzheimer’s disease (AD) and its development from different perspectives. Such information is closely related to the diagnosis, prevention, and treatment of AD, and hence it is necessary and critical to study AD through multi-modal data. Existing learning methods, however, usually ignore the influence of feature heterogeneity and directly fuse features in the last stages. Furthermore, most of these methods only focus on local fusion features or global fusion features, neglecting the complementariness of features at different levels and thus not sufficiently leveraging information embedded in multi-modal data. To overcome these shortcomings, we propose a novel framework for AD diagnosis that fuses gene, imaging, protein, and clinical data. Our framework learns feature representations under the same feature space for different modalities through a feature induction learning (FIL) module, thereby alleviating the impact of feature heterogeneity. Furthermore, in our framework, local and global salient multi-modal feature interaction information at different levels is extracted through a novel dual multilevel graph neural network (DMGNN). We extensively validate the proposed method on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset and experimental results demonstrate our method consistently outperforms other state-of-the-art multi-modal fusion methods. The code is publicly available on the GitHub website. (https://github.com/xiankantingqianxue/MIA-code.git)

Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models’ feasibility in clinical decision-making

BackgroundRecent enhancements in Large Language Models (LLMs) such as ChatGPT have exponentially increased user adoption. These models are accessible on mobile devices and support multimodal interactions, including conversations, code generation, and patient image uploads, broadening their utility in providing healthcare professionals with real-time support for clinical decision-making. Nevertheless, many authors have highlighted serious risks that may arise from the adoption of LLMs, principally related to safety and alignment with ethical guidelines. ObjectiveTo address these challenges, we introduce a novel methodological approach designed to assess the specific feasibility of adopting LLMs within a healthcare area, with a focus on clinical nursing, evaluating their performance and thereby directing their choice. Emphasizing LLMs’ adherence to scientific advancements, this approach prioritizes safety and care personalization, according to the “Organization for Economic Co-operation and Development” frameworks for responsible AI. Moreover, its dynamic nature is designed to adapt to future evolutions of LLMs. MethodThrough integrating advanced multidisciplinary knowledge, including Nursing Informatics, and aided by a prospective literature review, seven key domains and specific evaluation items were identified as follows:1.State of the Art Alignment & Safety.2.Focus, Accuracy & Management of Prompt Ambiguity.3.Data Integrity, Data Security, Ethics & Sustainability, in accordance with OECD Recommendations for Responsible AI.4.Temporal Variability of Responses (Consistency)5.Adaptation to specific standardized terminology and Classifications for healthcare professionals.6.General Capabilities: Post User Feedback Self-Evolution Capability and Organization in Chapters.7.Ability to Drive Evolution in Healthcare.A Peer Review by experts in Nursing and AI was performed, ensuring scientific rigor and breadth of insights for an essential, reproducible, and coherent methodological approach. By means of a 7-point Likert scale, thresholds are defined in order to classify LLMs as “unusable”, “usable with high caution”, and “recommended” categories.Nine state of the art LLMs were evaluated using this methodology in clinical oncology nursing decision-making, producing preliminary results. Gemini Advanced, Anthropic Claude 3 and ChatGPT 4 achieved the minimum score of the State of the Art Alignment & Safety domain for classification as “recommended”, being also endorsed across all domains. LLAMA 3 70B and ChatGPT 3.5 were classified as “usable with high caution.” Others were classified as unusable in this domain. ConclusionThe identification of a recommended LLM for a specific healthcare area, combined with its critical, prudent, and integrative use, can support healthcare professionals in decision-making processes.

Speech Cloning: Text-To-Speech Using VITS

Voice is one of the most common and natural communication methods for humans. Voice is becoming the primary interface for AI voice assistants like Amazon Alexa, as well as in autos and smart home devices. Homes and so on. As human-machine communication becomes more common, researchers are exploring technology that mimics genuine speech. Speech cloning is the practice of copying or mimicking another person's speech, usually utilizing modern technology and artificial intelligence (AI). This entails producing a synthetic or cloned version of someone's voice that sounds very similar to the actual speaker. The objective is to produce speech that is indistinguishable from the genuine person, both in tone and intonation. Instant Voice Cloning (IVC) in text-to-speech (TTS) synthesis refers to the TTS model's capacity to copy the voice of any reference speaker based on a short audio sample, without requiring extra speaker-specific training. This method is usually referred to as zero-shot TTS. IVC provides users with the flexibility to tailor the generated voice, offering significant value across diverse real-world applications. Examples include media content creation, personalized chatbots, and multi-modal interactions between humans and computers or extensive language models.

Effects of User Interface Orientation on Sense of Immersion in Augmented Reality

The sense of immersion provided by overlying computer-generated graphics over actual items is one of augmented reality’s distinguishing features. It is affected by multiple factors such as the user interface orientation in the virtual physical hybrid space. There have been many studies on realistic graphic rendering and multimodal interaction in augmented reality. Few studies, however, have systematically investigated the influence of the user interface orientations, more specifically the user-based orientation (UO) and object-based orientation (OO), on various aspects of immersion. We developed the UO and OO user interface prototypes and evaluated the sense of immersion in terms of engagement, engrossment, and overall immersion. The results showed that the user interface orientations in augmented reality had a significant influence on users’ perceived engagement and engrossment, whereas there was no significant difference in overall immersion. Particularly, compared to the OO user interfaces, the UO user interfaces exhibited higher perceived usability and slower focus of attention switching. Generalising implications for alternative augmented reality are provided.

Disclaiming knowledge to encourage participation in research group meetings

This study investigates instances where experienced researchers make explicit claims of lack of knowledge (e.g., ‘I have to reveal my ignorance completely’) in the context of paper discussions during research group meetings. Drawing on multimodal interaction analysis, the analysis focuses on epistemic disclaimers in their sequential contexts, and the local management of institutional identities and domains of knowledge. The analyzed data draw from video-ethnographic work involving participant observation and video recordings at a research program in Physical Chemistry at a Swedish university. Focusing on epistemic disclaimers occurring in first pair-parts, the analysis explores how senior researchers employ disclaimers to mark the eliciting function of their questions, as they work to encourage participation by opening the floor while positioning co-participants as (more) knowledgeable. The results evidence how the accomplishment of peer collaboration and knowledge distribution in scientific work involves the management of rights and responsibilities with respect to knowledge and scientific expertise.

A new XR-based human‐robot collaboration assembly system based on industrial metaverse

The industrial metaverse is an emerging paradigm of integrated information technologies. A human-robot collaboration assembly system is developed using extended reality (XR) and blockchain technologies. This system includes the physical, virtual reality (VR), augmented reality (AR) supporting, and blockchain systems. Virtual avatars represent physical operators and mobile robots in the XR environment. The VR system emulates real-time physical operations, facilitating pre-planning, online task optimization, and converting strategic decisions into executable instructions for physical robots. The AR system enhances the cognitive experience of intelligent operators, contributing to decision-making and VR system control through multi-modal interactions. The integration of the VR and AR systems in perception-decision-control enables real-time process optimization and the reverse control of physical operations. The system is implemented using a decentralized blockchain and end-edge-cloud collaborative network architecture to regulate interaction permissions for each scene characters, ensuring systematic functionality, robust communication, and comprehensive data management. A case study of a gearbox assembly is used to demonstrate the system’s development and efficacy. The results indicate the VR system's comprehensive perception and the AR system's detailed insights, yielding reliable information on global and local levels. The integration of artificial intelligence (AI) algorithms and spatiotemporal visualization during VR decision-making provides computational advantages. Incorporating human input into the AR system enables human-in-the-loop operational decision-making, providing optimal instructions to improve human-robot collaboration performance. Notably, the fusion of blockchain and end-edge-cloud architectures augments inter-system data exchange by efficiently managing and processing human-robot interactions and their diverse avatars. This system offers an innovative approach to human-robot collaboration assembly in manufacturing, with implications for analogous contexts.

Improving Error Correction and Text Editing Using Voice and Mouse Multimodal Interface

Improving Error Correction and Text Editing Using Voice and Mouse Multimodal Interface