Multimodal Human-machine Interaction Research Articles

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems

AbstractConventional Human–Machine Interaction (HMI) interfaces have predominantly relied on GUI and voice commands. However, natural human communication also consists of non-verbal communication, including hand gestures like pointing. Thus, recent works in HMI systems have tried to incorporate pointing gestures as an input, making significant progress in recognizing and integrating them with voice commands. However, existing approaches often treat these input modalities independently, limiting their capacity to handle complex multimodal instructions requiring intricate reasoning of language and gestures. On the other hand, multimodal tasks requiring complex reasoning are being challenged in the language and vision domain, but these typically do not include gestures like pointing. To bridge this gap, we explore one of the challenging multimodal tasks, called Referring Expression Comprehension (REC), within multimodal HMI systems incorporating pointing gestures. We present a virtual setup in which a robot shares an environment with a user and is tasked with identifying objects based on the user’s language and gestural instructions. Furthermore, to address this challenge, we propose a hybrid neuro-symbolic model combining deep learning’s versatility with symbolic reasoning’s interpretability. Our contributions include a challenging multimodal REC dataset for HMI systems, an interpretable neuro-symbolic model, and an assessment of its ability to generalize the reasoning to unseen environments, complemented by an in-depth qualitative analysis of the model’s inner workings.

High-Performance Hydrogel Sensors Enabled Multimodal and Accurate Human-Machine Interaction System for Active Rehabilitation.

Human-machine interaction (HMI) technology shows an important application prospect in rehabilitation medicine, but it is greatly limited by the unsatisfactory recognition accuracy and wearing comfort. Here, this work develops a fully flexible, conformable, and functionalized multimodal HMI interface consisting of hydrogel-based sensors and a self-designed flexible printed circuit board. Thanks to the component regulation and structural design of the hydrogel, both electromyogram (EMG) and forcemyography (FMG) signals can be collected accurately and stably, so that they are later decoded with the assistance of artificial intelligence (AI). Compared with traditional multichannel EMG signals, the multimodal human-machine interaction method based on the combination of EMG and FMG signals significantly improves the efficiency of human-machine interaction by increasing the information entropy of the interaction signals. The decoding accuracy of the interaction signals from only two channels for different gestures reaches 91.28%. The resulting AI-powered active rehabilitation system can control a pneumatic robotic glove to assist stroke patients in completing movements according to the recognized human motion intention. Moreover, this HMI interface is further generalized and applied to other remote sensing platforms, such as manipulators, intelligent cars, and drones, paving the way for the design of future intelligent robot systems.

On the Personalized Learning Space in Educational Metaverse Based on Heart Rate Signal

The application of the metaverse provides a new perspective for educational innovation and talent training in the new century. By creating multi-channel perceptions such as sight, hearing, and touch through the VR/AR/MR immersive environment, the study found that learners' emotional engagement and cognitive level can be effectively improved in the metaverse. The metaverse integrates information technology, intelligent technology, and digital technology to realize a multimodal human-machine interaction and intelligent evaluation. On this basis, a personalized learning space in the educational metaverse is developed, which is researched by the PPG signal of the learning process, the pre-test, and post-test data to verify each other. In addition, it is verified that learners' learning engagement and learning effects are affected in the personalized learning space of the educational metaverse. A comparison of PPG signal-based metaverse learning and real-world classroom learning is taken.

Developing a Multimodal HMI Design Framework for Automotive Wellness in Autonomous Vehicles

With the development of autonomous technology, the research into multimodal human-machine interaction (HMI) for autonomous vehicles (AVs) has attracted extensive attention, especially in automotive wellness. To support the design of HMIs for automotive wellness in AVs, this paper proposes a multimodal design framework. First, three elements of the framework were envisioned based on the typical composition of an interactive system. Second, a five-step process for utilizing the proposed framework was suggested. Third, the framework was applied in a design education course for exemplification. Finally, the AttrakDiff questionnaire was used to evaluate these interactive prototypes with 20 participants who had an affinity for HMI design. The questionnaire responses showed that the overall impression was positive and this framework can help design students to effectively identify research gaps and expand design concepts in a systematic way. The proposed framework offers a design approach for the development of multimodal HMIs for autonomous wellness in AVs.

Research on Effect of Load Stimulation Change on Heart Rate Variability of Women Volleyball Athletes

Objective. To explore the effect of different training load stimulation on heart rate variability level of Chinese elite female volleyball players. Through two-year follow-up experiment, this paper uses OmegaWave Sport Technology system to track and test the heart rate variability level and central nervous system parameters of 25 elite Chinese women volleyball players who participated in the national adult volleyball training in 2019 and 2020. It is found that the HRV time-domain index of the players under the stimulation of three stages of training load during the winter training in 2020 is determined. Frequency-domain index has significant influence on response stability of central nervous system. In order to further explore the influence of HRV on response stability of central nervous system, a feature classification method based on distance evaluation is proposed for experimental data processing. Through the multimodal human-machine interaction (M-HMI), advanced machine learning is used to promote the cooperative interaction between human and intelligent body. After analysis, SDNN and LF n.u. have a significant impact on the average reaction time. It shows that some indexes tested by the OmegaWave system can reflect the real-time physical function state of athletes sensitively and play an active role in diagnosis of fatigue of athletes' central nervous system. HRV time-domain and frequency-domain indexes, as parameters to evaluate the body functional state of excellent female volleyball players in the preparation process of competition, can sensitively reflect the level of autonomic nerve regulation of athletes in three different load stages.

On the Gap between Domestic Robotic Applications and Computational Intelligence

Aspired to build intelligent agents that can assist humans in daily life, researchers and engineers, both from academia and industry, have kept advancing the state-of-the-art in domestic robotics. With the rapid advancement of both hardware (e.g., high performance computing, smaller and cheaper sensors) and software (e.g., deep learning techniques and computational intelligence technologies), robotic products have become available to ordinary household users. For instance, domestic robots have assisted humans in various daily life scenarios to provide: (1) physical assistance such as floor vacuuming; (2) social assistance such as chatting; and (3) education and cognitive assistance such as offering partnerships. Crucial to the success of domestic robots is their ability to understand and carry out designated tasks from human users via natural and intuitive human-like interactions, because ordinary users usually have no expertise in robotics. To investigate whether and to what extent existing domestic robots can participate in intuitive and natural interactions, we survey existing domestic robots in terms of their interaction ability, and discuss the state-of-the-art research on multi-modal human–machine interaction from various domains, including natural language processing and multi-modal dialogue systems. We relate domestic robot application scenarios with state-of-the-art computational techniques of human–machine interaction, and discuss promising future directions towards building more reliable, capable and human-like domestic robots.

Multi-modal human-computer interaction system in cockpit

In order to explore new human-machine interaction methods, a set of multi-modal human-machine interaction coordinated control system is proposed, which realizes the basic flight control based on the change of the pilot’s field of view, touch control, voice control and other information obtained from multi-mode coordinated control. This system introduces a new type of human-computer interaction into the cockpit application on the basis of the human-computer interaction interface for flight driving, and has carried out research on multi-mode collaborative control system, mainly including eye movement interaction, touch interaction gesture Interaction and voice interaction. Finally, this project formed a multi-mode cooperative control software and hardware system.

Multi-modal human-machine interaction for human intelligence augmentation

Human-machine symbiosis and harmony is an ultimate goal of human-machine interaction (HMI). Recently, the rapid development of artificial intelligence has given rise to concern regarding the domination of machine intelligence over human intelligence. Enhancing human intelligence through multi-modal HMI technology is becoming an important research topic. The studies on neural plasticity indicate that the core capabilities of human cognition including attention control ability and working memory capacity could be enhanced and trained through several approaches including visual-auditory games, haptic interaction tasks, transcranial electromagnetic stimulation, and brain-computer interfaces. In this paper, we propose a systematic paradigm of augmenting cognition through HMI tasks by seamlessly integrating haptic-visual-aural multisensory feedback. This paradigm is constructed on analyzing the inherent features of virtual reality including immersion, interaction, and imagination. Based on the concept of cybernetics, cognitive enhancement methods based on Hebbian learning are proposed to achieve several goals including the realization of controllable cognitive loads, immediate physiological feedback, and bidirectional mind-body interaction. The proposed paradigm may provide new tools for revealing the mechanism of neural plasticity and promote the development of novel human-machine interaction devices; it is promising in the generation of practical values in the domain of personalized education, neural rehabilitation, and cognitive training of specialized personnel.

Multimodal speech and gesture control of AGVs, including EEG-based measurements of cognitive workload

Abstract Automated guided vehicles (AGVs) with autonomous behavior and decentralized human–machine interaction (HMI) are suitable for use in logistics. To facilitate natural interaction, HMI may involve both speech and gesture control. This paper presents a new cognitive approach based on electroencephalography (EEG) for multimodal HMI combining speech and gesture control for AGVs used in logistics. The results indicate that implicit EEG-based measures such as alertness and relaxation significantly affect speech control performance. Consequently, monitoring the user's cognitive workload during logistic operations may lead to a substantial improvement in work performance.

Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction

Quality of Service (QoS) and Quality of Experience (QoE) have to be considered when designing, building and maintaining services involving multimodal human–machine interaction. In order to guide the assessment and evaluation of such services, we first develop a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human–machine interactions. It consists of three layers: (1) The quality factors influencing QoS and QoE related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place within the user. For each of these layers, we then provide metrics which are able to capture the QoS and QoE aspects in a quantitative way, either via questionnaires or performance measures. The metrics are meant to guide system evaluation and make it more systematic and comparable.

Cognitive Principles in Robust Multimodal Interpretation

Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, automated interpretation of user multimodal inputs is important. Inspired by the previous investigation on cognitive status in multimodal human machine interaction, we have developed a greedy algorithm for interpreting user referring expressions (i.e., multimodal reference resolution). This algorithm incorporates the cognitive principles of Conversational Implicature and Givenness Hierarchy and applies constraints from various sources (e.g., temporal, semantic, and contextual) to resolve references. Our empirical results have shown the advantage of this algorithm in efficiently resolving a variety of user references. Because of its simplicity and generality, this approach has the potential to improve the robustness of multimodal input interpretation.

An approach to multi-modal human–machine interaction for intelligent service robots

The paper describes a multi-modal scheme for human–robot interaction suited for a wide range of intelligent service robot applications. Operating in un-engineered, cluttered, and crowded environments, such robots have to be able to actively contact potential users in their surroundings and to offer their services in an appropriate manner. Starting from a real application scenario, the usage of a robot as mobile information kiosk in a home store, some reliable methods for vision-based interaction, sound analysis and speech output have been developed. These methods are integrated into a prototypical interaction cycle that can be assumed as a general approach to human–machine interaction. Experimental results demonstrate the strengths and weaknesses of the proposed methods.

Handling Knowledge Sources in Human-Machine Interaction

This article describes the various knowledge sources that, in general, are required to handle multimodal human-machine interaction efficiently: these are called the task, user, dialogue, environment and system models. The first part discusses the content of these models. Special emphasis is given on problems that occur when speech is combined with other modalities. The second part focuses on spoken language characteristics and proposes an adapted semantic representation for the task model. It also describes a stochastic method to collect and process the information related to this model. The conclusion discusses an extension of such a stochastic method to multimodality.

Development Trends in Automation; Benefits to small- and medium-sized Manufacturing Enterprises

Automated guided vehicles (AGVs) with autonomous behavior and decentralized human–machine interaction (HMI) are suitable for use in logistics. To facilitate natural interaction, HMI may involve both speech and gesture control. This paper presents a new cognitive approach based on electroencephalography (EEG) for multimodal HMI combining speech and gesture control for AGVs used in logistics. The results indicate that implicit EEG-based measures such as alertness and relaxation significantly affect speech control performance. Consequently, monitoring the user's cognitive workload during logistic operations may lead to a substantial improvement in work performance.