Multimodal Interactive Systems Research Articles

Qualitative Research of the Multimodal In-Vehicle Interaction Systems Latency Perception

As the automotive industry evolves towards greater technological intelligence, multimodal in-vehicle interaction systems introduce new challenges, particularly in managing system latency-the delay between user input and system response. This study aims to explore how various factors influence latency perception in in-vehicle multimodal interactions, offering a comprehensive framework to guide future design improvements. Our research, conducted in three phases, began with mapping key input and output modalities through literature reviews and expert evaluations. We then examined real-world user experiences to identify specific latency-related issues in common interaction scenarios. Finally, focus groups refined a framework that categorizes tasks based on interaction complexity, sensitivity, and tolerance to latency. The framework highlights how different modalities impact user sensitivity to delays and their tolerance levels, providing valuable insights for designers. It emphasizes that tasks requiring higher cognitive engagement and quicker responses demand more attention to latency. Although focused on intelligent vehicles, the framework is applicable to traditional vehicles with similar systems and aligns with upcoming safety regulations. This research contributes to creating safer, more intuitive in-vehicle systems that enhance user experience and meet regulatory standards.

Augmented Reality for extremity hemorrhage training: a usability study.

Limb massive hemorrhage is the first cause of potentially preventable death in trauma. Its prompt and proper management is crucial to increase the survival rate. To handle a massive hemorrhage, it is important to train people without medical background, who might be the first responders in an emergency. Among the possible ways to train lay rescuers, healthcare simulation allows to practice in a safe and controlled environment. In particular, immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) give the possibility to provide real time feedback and present a realistic and engaging scenario, even though they often lack personalization. This work aims to overcome the above-mentioned limitation, by presenting the design, development and usability test of an AR application to train non-experienced users on the use of antihemorrhagic devices. The application combines a Microsoft Hololens2 headset, with an AR application developed in Unity Game Engine. It includes a training scenario with a multimodal interactive system made of visual and audio cues, that would adapt to user's learning pace and feedback preference. Usability tests on 20 subjects demonstrated that the system is well tolerated in terms of discomfort and workload. Also, the system has been high rated for usability, user experience, immersion and sense of presence. These preliminary results suggest that the combination of AR with multimodal cues can be a promising tool to improve hemorrhage management training, particularly for unexperienced users. In the future, the proposed application might increase the number of people who know how to use an anti-hemorrhagic device.

Talk to the Wall: The Role of Speech Interaction in Collaborative Visual Analytics.

We present the results of an exploratory study on how pairs interact with speech commands and touch gestures on a wall-sized display during a collaborative sensemaking task. Previous work has shown that speech commands, alone or in combination with other input modalities, can support visual data exploration by individuals. However, it is still unknown whether and how speech commands can be used in collaboration, and for what tasks. To answer these questions, we developed a functioning prototype that we used as a technology probe. We conducted an in-depth exploratory study with 10 participant pairs to analyze their interaction choices, the interplay between the input modalities, and their collaboration. While touch was the most used modality, we found that participants preferred speech commands for global operations, used them for distant interaction, and that speech interaction contributed to the awareness of the partner's actions. Furthermore, the likelihood of using speech commands during collaboration was related to the personality trait of agreeableness. Regarding collaboration styles, participants interacted with speech equally often whether they were in loosely or closely coupled collaboration. While the partners stood closer to each other during close collaboration, they did not distance themselves to use speech commands. From our fndings, we derive and contribute a set of design considerations for collaborative and multimodal interactive data analysis systems. All supplemental materials are available at https://osf.io/8gpv2.

Intent Prediction in AR Shopping Experiences Using Multimodal Interactions of Voice, Gesture, and Eye Tracking: A Machine Learning Perspective

Augmented Reality (AR) is revolutionizing the shopping experience by allowing consumers to interact with virtual products in real-time. Intent prediction – the mechanism of predicting a consumer’s intention based on their behavioral patterns and actions – is crucial for enhancing the personalization of AR shopping environments. This paper explores how multimodal interactions, including voice commands, gesture recognition, and eye tracking, can be integrated into AR shopping experiences to predict user intent more effectively. We review current advancements in multimodal interaction systems, discuss the importance of intent prediction in AR, and assess the impact of combining multiple input modalities on prediction accuracy. Our research identifies the challenges and future directions for intent prediction in AR shopping landscapes, aiming to improve user engagement, personalization, and the overall shopping experience.

DDC-Chat: Achieving accurate distracted driver classification through instruction tuning of visual language model

Driver behavior is a critical factor in road safety, highlighting the need for advanced methods in Distracted Driving Classification (DDC). In this study, we introduce DDC-Chat, a novel classification method based on a Visual large Language Model (VLM). DDC-Chat is an interactive multimodal system built upon LLAVA-Plus, fine-tuned specifically for addressing distracted driving detection. It utilizes logical reasoning chains to activate visual skills, including segmentation and pose detection, through end-to-end training. Furthermore, instruction tuning allows DDC-Chat to continuously incorporate new visual skills, enhancing its ability to classify distracted driving behavior. Our extensive experiments demonstrate that DDC-Chat achieves state-of-the-art performance on public DDC datasets, surpassing previous benchmarks. In evaluations on the 100-Driver dataset, the model exhibits superior results in both zero-shot and few-shot learning contexts, establishing it as a valuable tool for improving driving safety by accurately identifying driver distraction. Due to the computational intensity of inference, DDC-Chat is optimized for deployment on remote servers, with data streamed from in-vehicle monitoring systems for real-time analysis. More details are available on the project homepage: https://LCP-DDC-Chat.github.io/.

E-Learning system application in art entrepreneurship teaching based on multimodal feature fusion and neural network

College student entrepreneurship teams can leverage the advantages of mobile internet resource integration and online collaborative production platforms to easily integrate various resources, thereby achieving rapid start-up and development of entrepreneurial projects. This also makes the threshold for college students to start businesses lower and easier to achieve. This article designs an interactive aesthetic education and entrepreneurship teaching system based on multimodal feature fusion and interactive systems. The core concept of the interactive aesthetic education and entrepreneurship teaching system is multimodal feature fusion and interactive system. Multimodal feature fusion can integrate different teaching resources, so that students can better understand the factors that need to be considered in the entrepreneurial process and practice. Interactive systems can enable closer interaction and communication between students and teachers, thereby better understanding and mastering entrepreneurial knowledge and skills. The design and implementation of a systematic system will help enhance the entrepreneurial ability and quality of college students, help them better understand the entrepreneurial process and factors that need to be considered in practice, and provide entrepreneurial practice reference for students in Chinese universities who have the intention of designing entrepreneurship.

Personalised Multi-modal Interactive Recommendation with Hierarchical State Representations

Multi-modal interactive recommender systems (MMIRS) can effectively guide users towards their desired items through multi-turn interactions by leveraging the users’ real-time feedback (in the form of natural-language critiques) on previously recommended items (such as images of fashion products). In this scenario, the users’ preferences can be expressed by both the users’ past interests from their historical interactions and their current needs from the real-time interactions. However, it is typically challenging to make satisfactory personalised recommendations across multi-turn interactions due to the difficulty in balancing the users’ past interests and the current needs for generating the users’ state (i.e., current preferences) representations over time. However, hierarchical reinforcement learning has been successfully applied in various fields by decomposing a complex task into a hierarchy of more easily addressed subtasks. In this journal article, we propose a novel personalised multi-modal interactive recommendation model (PMMIR) using hierarchical reinforcement learning to more effectively incorporate the users’ preferences from both their past and real-time interactions. In particular, PMMIR decomposes the personalised interactive recommendation process into a sequence of two subtasks with hierarchical state representations: a first subtask where a history encoder learns the users’ past interests with the hidden states of history for providing personalised initial recommendations and a second subtask where a state tracker estimates the current needs with the real-time estimated states for updating the subsequent recommendations. The history encoder and the state tracker are jointly optimised with a single objective by maximising the users’ future satisfaction with the recommendations. Following previous work, we train and evaluate our PMMIR model using a user simulator that can generate natural-language critiques about the recommendations as a surrogate for real human users. Experiments conducted on two derived fashion datasets from two well-known public datasets demonstrate that our proposed PMMIR model yields significant improvements in comparison to the existing state-of-the-art baseline models. The datasets and code are publicly available at: https://github.com/yashonwu/pmmir

Application of multimodality perception scene construction based on Internet of Things (IoT) technology in art teaching

Numerous impediments beset contemporary art education, notably the unidimensional delivery of content and the absence of real-time interaction during instructional sessions. This study endeavors to surmount these challenges by devising a multimodal perception system entrenched in Internet of Things (IoT) technology. This system captures students’ visual imagery, vocalizations, spatial orientation, movements, ambient luminosity, and contextual data by harnessing an array of interaction modalities encompassing visual, auditory, tactile, and olfactory sensors. The synthesis of this manifold information about learning scenarios entails strategically placing sensors within physical environments to facilitate intuitive and seamless interactions. Utilizing digital art flower cultivation as a quintessential illustration, this investigation formulates tasks imbued with multisensory channel interactions, pushing the boundaries of technological advancement. It pioneers advancements in critical domains such as visual feature extraction by utilizing DenseNet networks and voice feature extraction leveraging SoundNet convolutional neural networks. This innovative paradigm establishes a novel art pedagogical framework, accentuating the importance of visual stimuli while enlisting other senses as complementary contributors. Subsequent evaluation of the usability of the multimodal perceptual interaction system reveals a remarkable task recognition accuracy of 96.15% through the amalgamation of Mel-frequency cepstral coefficients (MFCC) speech features with a long-short-term memory (LSTM) classifier model, accompanied by an average response time of merely 6.453 seconds—significantly outperforming comparable models. The system notably enhances experiential fidelity, realism, interactivity, and content depth, ameliorating the limitations inherent in solitary sensory interactions. This augmentation markedly elevates the caliber of art pedagogy and augments learning efficacy, thereby effectuating an optimization of art education.

Optimization Analysis of AI Intelligent English Teaching Strategies Based on Computer Virtual Reality Technology

The Internet of Things and other technology breakthroughs have a big impact on how English is taught (IoTV). The investigation of the actual English teaching process and the identification of student features come first. In order to use the interpolation technique with the student image-based feature detection method, we investigate the IoTV reform. As a result, we find a clever algorithm for IoTV that recognises student features more effectively. Artificial intelligence (AI) is the design technique used to create the intelligence algorithm for pupil feature recognition. A comprehensive multifunctional human-computer interaction system utilizes a variety of input and output streams. In addition to the standard computer keyboard, cursor clicking, and screen touching, the most recent speech and facial recognition technology can be employed for data input. Students learned to orally interact with the robot and act as a guide to various destinations. The Multimodal Interaction System for English Education (MMIEE) is an investigation into the use of network and artificial intelligence (AI) in teaching. Recurrent neural network (Rein RNN)-based Reinforced learning is utilized for the perpectual evaluation model to categorize the questions posed by teachers in terms of their content and kind, and to conduct experimental investigation. The results show that the proposed Rein_RNN model achieves 98.6% of accuracy in 4.3sec of mean evaluation time.

Development and validation of an art-inspired multimodal interactive technology system for a multi-component intervention for older people: a pilot study

IntroductionThe World Health Organization (WHO) acknowledges the presence of a significant body of research on the positive effects of the arts on health, considering a variety of factors including physical well-being, quality of life, and social and community impact. The model that underlies cultural welfare puts the performing arts, visual arts, and cultural heritage at the service of people personal and societal well-being. The potential connections between movements of the body and artistic content have been extensively studied over time, considering movement as a non-verbal language with a universal character.MethodsThis pilot study presents the results of the validation of an innovative multimodal system, the DanzArTe-Emotional Wellbeing Technology, designed to support active and participative experience of older people providing physical and cognitive activation through a full-body physical interaction with a traditional visual work of art of religious subject. DanzArTe supports a replicable treatment protocol for multidimensional frailty, administered through a low cost and scalable technological platform capable of generating real-time visual and auditory feedback (interactive sonification) from the automated analysis of individual as well as joint movement expressive qualities. The study involved 45 participants, 23 of whom participated in the DanzArTe program and 22 who were included in the control group.ResultsThe two groups were similar in terms of age (p = 0.465) and gender (p = 0.683). The results showed that the DanzArTe program had a positive impact on participants' self-perceived psychological health and well-being (Mean Psychological General Well-Being Index—Short T1 = 19.6 ± 4.3 Vs. T2 = 20.8 ± 4.9; p = 0.029). The same trend was not observed in the control group (p = 0.389).DiscussionThe findings suggest that such programs may have a significant impact particularly on the mental and social well-being of older adults and could be a valuable tool for promoting healthy aging and improving quality of life.

MusicARLtrans Net: a multimodal agent interactive music education system driven via reinforcement learning.

In recent years, with the rapid development of artificial intelligence technology, the field of music education has begun to explore new teaching models. Traditional music education research methods have primarily focused on single-modal studies such as note recognition and instrument performance techniques, often overlooking the importance of multimodal data integration and interactive teaching. Existing methods often struggle with handling multimodal data effectively, unable to fully utilize visual, auditory, and textual information for comprehensive analysis, which limits the effectiveness of teaching. To address these challenges, this project introduces MusicARLtrans Net, a multimodal interactive music education agent system driven by reinforcement learning. The system integrates Speech-to-Text (STT) technology to achieve accurate transcription of user voice commands, utilizes the ALBEF (Align Before Fuse) model for aligning and integrating multimodal data, and applies reinforcement learning to optimize teaching strategies. This approach provides a personalized and real-time feedback interactive learning experience by effectively combining auditory, visual, and textual information. The system collects and annotates multimodal data related to music education, trains and integrates various modules, and ultimately delivers an efficient and intelligent music education agent. Experimental results demonstrate that MusicARLtrans Net significantly outperforms traditional methods, achieving an accuracy of 96.77% on the LibriSpeech dataset and 97.55% on the MS COCO dataset, with marked improvements in recall, F1 score, and AUC metrics. These results highlight the system's superiority in speech recognition accuracy, multimodal data understanding, and teaching strategy optimization, which together lead to enhanced learning outcomes and user satisfaction. The findings hold substantial academic and practical significance, demonstrating the potential of advanced AI-driven systems in revolutionizing music education.

A Parallel Multimodal Integration Framework and Application for Cake Shopping

Multimodal interaction systems can provide users with natural and compelling interactive experiences. Despite the availability of various sensing devices, only some commercial multimodal applications are available. One reason may be the need for a more efficient framework for fusing heterogeneous data and addressing resource pressure. This paper presents a parallel multimodal integration framework that ensures that the errors and external damages of integrated devices remain uncorrelated. The proposed relative weighted fusion method and modality delay strategy process the heterogeneous data at the decision level. The parallel modality operation flow allows each device to operate across multiple terminals, reducing resource demands on a single computer. The universal fusion methods and independent devices further remove constraints on the integrated modality number, providing the framework with extensibility. Based on the framework, we develop a multimodal virtual shopping system, integrating five input modalities and three output modalities. The objective experiments show that the system can accurately fuse heterogeneous data and understand interaction intent. User studies indicate the immersive and entertaining of multimodal shopping. Our framework proposes a development paradigm for multimodal systems, fostering multimodal applications across various domains.

High-Performance Hydrogel Sensors Enabled Multimodal and Accurate Human-Machine Interaction System for Active Rehabilitation.

Human-machine interaction (HMI) technology shows an important application prospect in rehabilitation medicine, but it is greatly limited by the unsatisfactory recognition accuracy and wearing comfort. Here, this work develops a fully flexible, conformable, and functionalized multimodal HMI interface consisting of hydrogel-based sensors and a self-designed flexible printed circuit board. Thanks to the component regulation and structural design of the hydrogel, both electromyogram (EMG) and forcemyography (FMG) signals can be collected accurately and stably, so that they are later decoded with the assistance of artificial intelligence (AI). Compared with traditional multichannel EMG signals, the multimodal human-machine interaction method based on the combination of EMG and FMG signals significantly improves the efficiency of human-machine interaction by increasing the information entropy of the interaction signals. The decoding accuracy of the interaction signals from only two channels for different gestures reaches 91.28%. The resulting AI-powered active rehabilitation system can control a pneumatic robotic glove to assist stroke patients in completing movements according to the recognized human motion intention. Moreover, this HMI interface is further generalized and applied to other remote sensing platforms, such as manipulators, intelligent cars, and drones, paving the way for the design of future intelligent robot systems.

Multimodal intent understanding and interaction system for elderly-assisted companionship

Multimodal intent understanding and interaction system for elderly-assisted companionship

Deep neural networks for multimodal perception and human-computer interaction technology in art design

Abstract The first part of this paper examines the aesthetic and application advantages of art design using human-computer interaction technology and develops a multimodal perceptual human-computer interaction system for art design. Multimodal data is obtained using multi-scale convolutional kernels for acoustic feature extraction and deep convolutional neural networks for multiple interaction image feature fusion. Finally, a test analysis is conducted to verify the system's effectiveness in this paper. According to the results, the system has an average wake-up success rate of 99.51% and a wake-up response time of 0.3665 seconds. Implementing human-computer interaction technology and deep neural networks in art design is effective and promotes the development of art design.

MyHealthHub for older adult inpatients to reduce loneliness, and improve patient engagement and mental health: protocol of a pilot randomized controlled trial

Objectives Older Canadian adults make up 85% of hospital stays which are associated with increased loneliness, stress, anxiety, and/or depression. There is a need for novel approaches to reduce loneliness and mental health outcomes in older adult hospital inpatients to prevent further strain on an already overwhelmed healthcare system. Methods This is a pilot randomized controlled trial (RCT) exploring the efficacy of a bedside multimodal interaction system, myHealthHub, on loneliness, quality of life (QOL), patient engagement, and other mental health outcomes compared to an active control group in older adult inpatients (n = 60) from baseline to 5-days. Qualitative analyses will be conducted through semi-structured interviews with older adults (n = 8-10) and hospital staff, nurses, and clinicians (n = 4-5) facilitating the service to evaluate patient engagement and experience with myHealthHub. Results Not applicable. Conclusion This novel pilot clinical trial will obtain preliminary data on the efficacy of myHealthHub in reducing loneliness, QOL, patient engagement, and mental health outcomes in older adult inpatients. If successful, this could provide a potential means to improve patient experience in hospitals and reduce the burden and additional expense on the healthcare system.

A multimodal domestic service robot interaction system for people with declined abilities to express themselves

Driven by the shortage of qualified nurses and the increasing average age of the population, the ambient assisted living style using intelligent service robots and smart home systems has become an excellent choice to free up caregiver time and energy and provide users with a sense of independence. However, users’ unique environments and differences in abilities to express themselves through different interaction modalities make intention recognition and interaction between user and service system very difficult, limiting the use of these new nursing technologies. This paper presents a multimodal domestic service robot interaction system and proposes a multimodal fusion algorithm for intention recognition to deal with these problems. The impacts of short-term and long-term changes were taken into account. Implemented interaction modalities include touch, voice, myoelectricity gesture, visual gesture, and haptics. Users could freely choose one or more modalities through which to express themselves. Virtual games and virtual activities of independent living were designed for pre-training and evaluating users’ abilities to use different interaction modalities in their unique environments. A domestic service robot interaction system was built, on which a set of experiments were carried out to test the system’s stability and intention recognition ability in different scenarios. The experiment results show that the system is stable and effective and can adapt to different scenarios. In addition, the intention recognition rate in the experiments was 93.62%. Older adults could master the system quickly and use it to provide some assistance for their independent living.

Research on Multimodal Interaction System of Lower Limb Exoskeleton Oriented to Competitive Training

Research on Multimodal Interaction System of Lower Limb Exoskeleton Oriented to Competitive Training

ConvXAI: a System for Multimodal Interaction with Any Black-box Explainer

ConvXAI: a System for Multimodal Interaction with Any Black-box Explainer

Investigating Methods for Cognitive Workload Estimation for Assistive Robots.

Robots interacting with humans in assistive contexts have to be sensitive to human cognitive states to be able to provide help when it is needed and not overburden the human when the human is busy. Yet, it is currently still unclear which sensing modality might allow robots to derive the best evidence of human workload. In this work, we analyzed and modeled data from a multi-modal simulated driving study specifically designed to evaluate different levels of cognitive workload induced by various secondary tasks such as dialogue interactions and braking events in addition to the primary driving task. Specifically, we performed statistical analyses of various physiological signals including eye gaze, electroencephalography, and arterial blood pressure from the healthy volunteers and utilized several machine learning methodologies including k-nearest neighbor, naive Bayes, random forest, support-vector machines, and neural network-based models to infer human cognitive workload levels. Our analyses provide evidence for eye gaze being the best physiological indicator of human cognitive workload, even when multiple signals are combined. Specifically, the highest accuracy (in %) of binary workload classification based on eye gaze signals is 80.45 ∓ 3.15 achieved by using support-vector machines, while the highest accuracy combining eye gaze and electroencephalography is only 77.08 ∓ 3.22 achieved by a neural network-based model. Our findings are important for future efforts of real-time workload estimation in the multimodal human-robot interactive systems given that eye gaze is easy to collect and process and less susceptible to noise artifacts compared to other physiological signal modalities.