Multimodal Human-computer Interaction Research Articles

Modelling conversational agent with empathy mechanism

Empathy mechanism in communication is the cornerstone for effective and meaningful interaction. Establishing an empathy mechanism in conversation agent (CA) requires accurate recognition of users’ emotions to facilitate generates appropriate empathetic responses. Therefore, we proposed a Multimodal Emotion Recognition Model (MERM) to recognizes a user’s emotional state from multimodal data (audio, facial expressions, and conversation text) during conversation, and an Interactive Empathetic Conversation Model (IECM) to generate empathetic responses based on the MERM. Comparative and ablation study results indicated that the proposed models outperform existing methods in recognizing the user’s emotions and generating appropriate empathetic responses. We also conducted an experiment study, the results indicated that the CA significantly enhances the user’s emotional experience.

Perception and Decision-Making for Multi‑Modal Interaction Based on Fuzzy Theory in the Dynamic Environment

The projection-augmented reality assembly assistance system, which supports natural multi-modal human–computer interaction, has the potential to improve interaction efficiency. However, the effectiveness of the interaction method is compromised by the interference caused by the dynamically changing industrial environment. To address this issue, this study proposed a multi-model interaction model based on fuzzy theory and developed an environment perception decision-making interaction system. This system continuously monitors real-time environment data, evaluates interaction performance, and visually recommends the optimal method to assist users in completing their tasks. A comparative experiment was conducted between the proposed system and a non-environment perception interaction system. The collected data, including interaction success rate, task time, system usability, and user experience, were analyzed. The results demonstrated that the proposed system effectively enhances the adaptability and comfort of computer systems in complex environments and improves communication efficiency between humans and machines.

Harnessing Power of Multimodal Interaction, their Challenges and Future Prospect – A Review

Multimodal interaction refers to the combination of smart speakers and displays. It gives users the option to engage with various input and output modalities. When interacting with other individuals, humans use more nonverbal cues compared to verbal cues. They communicate with each other using a variety of modalities, including gestures, eye contact, and facial expressions. This type of communication is known as multimodal interaction. A specific type of multimodal interaction called human-computer interaction (HCI) makes it easier for people to communicate with machines. Several studies employing the aforementioned numerous modalities will discover that machines could quickly interact with a person by disclosing their feelings or actions. The research presented here provides an in-depth overview of multimodal interaction, HCI, the difficulties and advancements encountered in this field, and its prospects for future technological improvement.

A MULTI-MODAL INTENTION INTEGRATION METHOD FOR UNMANNED AERIAL VEHICLES BASED ON THE COMBINATION OF BIOELECTRICAL SIGNALS AND CONVENTIONAL HUMAN–COMPUTER INTERACTION

With the rapid development of intelligent information technology, human–computer interaction for unmanned aerial vehicle (UAV) systems has gradually developed from single-mode interaction to multi-modal interaction. Due to the problems of information ambiguity and operational redundancy in multi-modal interaction, and the inability to effectively realize the coupling of human–computer interaction of different modalities, a priority strategy for multi-modal interaction for UAV systems is proposed. Living cells or tissues, whether in a static or active state, will produce regular electrical phenomena called bioelectricity. Human–computer interaction intention recognition technology based on bioelectrical signals can significantly enhance the real-time performance and flexibility of human–computer interaction and has great development potential. The characteristics of human–computer interaction intention recognition methods based on bioelectrical signals, such as electroencephalograms (EEGs), electromyograms (EMGs), electrooculograms (EOGs), and other conventional methods are introduced and analyzed, and then the basic characteristics of single-mode interaction and multi-modal interaction are compared. Furthermore, the priority strategies for UAV systems under different interaction modes are studied based on the analysis of task value evaluation parameters. The cumulative value, urgency, and task complexity are comprehensively analyzed and evaluated, and a set of priority strategies for multi-modal interaction is integrated. Through the simulation and verification of the multi-modal human–computer interaction software of the UAV system, it is determined that the proposed priority strategy conforms to the operation intention of the user, which improves the interaction efficiency and the reliability. In noisy environments, the UAV system control can be combined with the complementary information of various interaction methods to achieve precise control, which has obvious advantages.

A Multimodal Human-Computer Interaction for Smart Learning System

The rise of digitalization and computing devices has transformed the educational landscape, making traditional teaching methods less productive. In this context, early and continuous user interaction is crucial for designing and developing effective learning applications. The field of Human-Computer Interaction (HCI) has seen significant technological growth, enabling educators to provide quality educational services through smart input and output channels. However, to prevent students from discontinuing their studies and help them grow their careers, a multimodal HCI approach is needed. This paper proposes a multimodal deep learning multi-layer Convolutional Neural Network (CNN) to improve the educational experience. Our designed system aims to create a promising solution for improving the educational experience and enabling educators to provide high-quality educational services to students. Our implementation results show promising real-time performances, including a high success rate in a constriction learning concept, a quality interaction experience, and enhanced educational services. We evaluated the accuracy of five multimodal inputs, including Finger Touch (FT), Hands Up (HU), Hands Down (HD), Voice Command (VC), and Click/Typing (CT). The results indicate an average accuracy of 90.8%, 87%, 88.6%, 91.8%, and 87%, respectively, demonstrating the effectiveness of our proposed approach.

Predicting information usefulness in health information identification from modal behaviors

Finding useful health information should be the highest priority when identifying health information. Predicting information usefulness will significantly improve the effectiveness and efficiency of health information identification, which plays a vital role in fighting against misinformation. Modal behaviors, such as gesture and gaze, are promising indicators of usefulness since they deliver a reliable, thorough, natural, and direct process of user cognitive processing. Therefore, this study aimed to use gesture and gaze behaviors to predict whether information is useful for health information identification. Twenty-four college students were recruited to freely search for information using a smartphone to identify the truthfulness of four propositions (two were true and two were false) about public health epidemics. The participants' gesture behavior, gaze behavior, and information usefulness as perceived by themselves were collected. Based on user cognition, the process of information usefulness judgment was placed into two phases: skimming and reading. Thirty-one features derived from modal behaviors in each phase were extracted. Feature optimization based on the Mann-Whitney U test and random forest was performed. Five common algorithms were used to construct information usefulness prediction models, and these models were compared by the F1_score. Finally, dwell time and gaze entropy in the reading phase were the most important gesture and gaze features respectively. BP neural network was selected to build a unimodal model based on gesture, and gradient boosting decision tree was selected to build a unimodal model based on gaze and a multimodal model combining both. These models all achieved F1_score above 77% and were applicable to different scenarios in health information identification. The model based on gesture could satisfy strong technology or legal constrains, the model based on gaze was ideal for AR, MR or metaverse applications, and the model combining both offered an alternative for multimodal human-computer interaction.

Magnetic Motion Tracking for Natural Human Computer Interaction: A Review

In the field of multimodal human–computer interaction (HCI), hand (and finger) motion tracking remains a critical challenge because it is the input means of visual reproduction and provides the necessary parameters for real-time modeling of the haptic model. Among motion tracking methods including optical tracking methods, inertial tracking methods, and magnetic motion tracking (MMT) methods, the MMT method has a great development prospect as a marker-based method because of its advantages of no line-of-sight problem, being natural-oriented, and high accuracy. In this article, we first discuss the performance and application scenarios under different configurations of electromagnet-based MMT (EMMT) and permanent-magnet-based MMT (PMMT) systems. And then, the MMT algorithms are reviewed, which can be divided into two broad categories: The model-based algorithms include linear algorithm, nonlinear optimization algorithm, and recursive Bayesian algorithm, and the offline-data-based algorithms include neural network algorithm and lookup table (LUT) algorithm. In addition, a state-of-the-art comparison of MMT algorithms is given. In the end, we have an insightful discussion and give the expected future outlook of MMT.

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Multimodal human–computer interaction (HCI) systems pledge a more human–human-like interaction between machines and humans. Their prowess in emanating an unambiguous information exchange between the two makes these systems more reliable, efficient, less error prone, and capable of solving complex tasks. Emotion recognition is a realm of HCI that follows multimodality to achieve accurate and natural results. The prodigious use of affective identification in e-learning, marketing, security, health sciences, etc., has increased demand for high-precision emotion recognition systems. Machine learning (ML) is getting its feet wet to ameliorate the process by tweaking the architectures or wielding high-quality databases (DB). This paper presents a survey of such DBs that are being used to develop multimodal emotion recognition (MER) systems. The survey illustrates the DBs that contain multi-channel data, such as facial expressions, speech, physiological signals, body movements, gestures, and lexical features. Few unimodal DBs are also discussed that work in conjunction with other DBs for affect recognition. Further, VIRI, a new DB of visible and infrared (IR) images of subjects expressing five emotions in an uncontrolled, real-world environment, is presented. A rationale for the superiority of the presented corpus over the existing ones is instituted.

Development of an Integrated Haptic Sensor System for Multimodal Human–Computer Interaction Using Ultrasonic Array and Cable Robot

This paper presents the design and development of a novel haptic sensor system. It integrates cable-driven force and ultrasonic tactile feedback, which can produce multimodal haptic stimuli. The sensing element includes a Leapmotion, an ultrasonic transducer array, tension sensors, and rotary encoders, which are used to capture hand posture, project tactile points, measure cable force and length, respectively. Firstly, a 6-DOF cable-driven force feedback apparatus based on parallel mechanism is designed and ultrasonic phased array is combined to form a multimodal haptic feedback system. Secondly, a multimodal haptic fusion method for cable-driven force and ultrasonic tactile is firstly proposed to invoke realistic compound haptic sensations. To enhance the rendering effects of each subsystem, admittance control is developed for a cable robot, and a new perceived magnitude model is established for ultrasound tactile rendering. A psychophysical experiment is conducted to study the perceived characteristics of multimodal haptic stimuli. To verify the proposed system, a series of experiments were carried out, whose results indicated that the system performs well at multi-property haptic rendering and confirm the accuracy and sensitive advantage of our system in virtual reality applications. The results of our study indicate that this device has great application potential in human-computer interaction.

Systematic Review of Multimodal Human–Computer Interaction

This document presents a systematic review of Multimodal Human–Computer Interaction. It shows how different types of interaction technologies (virtual reality (VR) and augmented reality, force and vibration feedback devices (haptics), and tracking) are used in different domains (concepts, medicine, physics, human factors/user experience design, transportation, cultural heritage, and industry). A systematic literature search was conducted identifying 406 articles initially. From these articles, we selected 112 research works that we consider most relevant for the content of this article. The articles were analyzed in-depth from the viewpoint of temporal patterns, frequency of usage in types of technology in different domains, and cluster analysis. The analysis allowed us to answer relevant questions in searching for the next steps in work related to multimodal HCI. We looked at the typical technology type, how the technology type and frequency have changed in time over each domain, and how papers are grouped across metrics given their similarities. This analysis determined that VR and haptics are the most widely used in all domains. While VR is the most used, haptic interaction is presented in an increasing number of applications, suggesting future work on applications that configure VR and haptic together.

Multi-modal human-computer interaction system in cockpit

In order to explore new human-machine interaction methods, a set of multi-modal human-machine interaction coordinated control system is proposed, which realizes the basic flight control based on the change of the pilot’s field of view, touch control, voice control and other information obtained from multi-mode coordinated control. This system introduces a new type of human-computer interaction into the cockpit application on the basis of the human-computer interaction interface for flight driving, and has carried out research on multi-mode collaborative control system, mainly including eye movement interaction, touch interaction gesture Interaction and voice interaction. Finally, this project formed a multi-mode cooperative control software and hardware system.

INSMA: An integrated system for multimodal data acquisition and analysis in the intensive care unit.

Modern intensive care units (ICU) are equipped with a variety of different medical devices to monitor the physiological status of patients. These devices can generate large amounts of multimodal data daily that include physiological waveform signals (arterial blood pressure, electrocardiogram, respiration), patient alarm messages, numeric vitals data, etc. In order to provide opportunities for increasingly improved patient care, it is necessary to develop an effective data acquisition and analysis system that can assist clinicians and provide decision support at the patient bedside. Previous research has discussed various data collection methods, but a comprehensive solution for bedside data acquisition to analysis has not been achieved. In this paper, we proposed a multimodal data acquisition and analysis system called INSMA, with the ability to acquire, store, process, and visualize multiple types of data from the Philips IntelliVue patient monitor. We also discuss how the acquired data can be used for patient state tracking. INSMA is being tested in the ICU at University Hospitals Cleveland Medical Center.

Framing the Design Space of Multimodal Mid-Air Gesture and Speech-Based Interaction With Mobile Devices for Older People

Multimodal human–computer interaction has been sought to provide not only more compelling interactive experiences, but also more accessible interfaces to mobile devices. With the advance in mobile technology and in affordable sensors, multimodal research that leverages and combines multiple interaction modalities (such as speech, touch, vision, and gesture) has become more and more prominent. This article provides a framework for the key aspects in mid-air gesture and speech-based interaction for older adults. It explores the literature on multimodal interaction and older adults as technology users and summarises the main findings for this type of users. Building on these findings, a number of crucial factors to take into consideration when designing multimodal mobile technology for older adults are described. The aim of this work is to promote the usefulness and potential of multimodal technologies based on mid-air gestures and voice input for making older adults' interaction with mobile devices more accessible and inclusive.

Multimodal data fusion algorithm applied to robots

In recent years, the use of multimodal human-computer interaction technology to achieve the enhancement of human intelligence has become a new topic in human-computer interaction research. When the robot can’t react correctly in a single mode, it is necessary to realize multimodal fusion. To this end, this paper proposes a multimodal fusion algorithm that applies the data obtained by the CNN feature layer to the decision-level. The speech recognition text is semantically matched with the text in the text library, and the similar probability vector is returned. At the same time, the similarity probability vector of the gesture recognition is obtained, and the data is filtered by the threshold, and the set of high probability data codes is assigned to the two modes. The intersection operation, and the final instruction is sent to the robot. The experimental results show that the influence of environmental factors on the single channel result is reduced, and the single mode ambiguity problem is eliminated. The multi-channel fusion algorithm with additional weight is more accurate than the common multi-channel fusion algorithm. At the same time, it has also been well received by many test users.

A novel gaze-supported multimodal human-computer interaction for ultrasound machines.

Conventional ultrasound (US) machines employ a physical control panel (PCP) as the primary user interface for machine control. This panel is adjacent to the main machine display that requires the operator's constant attention. The switch of attention to the control panel can lead to interruptions in the flow of the medical examination. Some ultraportable machines also lack many physical controls. Furthermore, the need to both control the US machine and observe the US image may lead the practitioners to adopt unergonomic postures and repetitive motions that can lead to work-related injuries. Therefore, there is a need for a more efficient human-computer interaction method on US machines. To tackle some of the limitations with the PCP, we propose to merge the PCP into the main screen of the US machines. We propose to use gaze tracking and a handheld controller so that machine control can be achieved via a multimodal human-computer interaction (HCI) method that does not require one to touch the screen or look away from the US image. As a first step, a pop-up menu and measurement tool were designed on top of the US image based on gaze position for efficient machine control. A comparative study was performed on the BK Medical SonixTOUCH US machine. Participants were asked to complete the task of measuring the area of an ellipse-shaped tumor in a phantom using our gaze-supported HCI method as well as the traditional method. The user study indicates that the task completion time can be reduced by [Formula: see text] when using our gaze-supported HCI, while no extra workload is imposed on the operators. Our preliminary study suggests that, when combined with a simple handheld controller, eye gaze tracking can be integrated into the US machine HCI for more efficient machine control.

Data fusion methods in multimodal human computer dialog

In multimodal human computer dialog, non-verbal channels, such as facial expression, posture, gesture, etc, combined with spoken information, are also important in the procedure of dialogue. Nowadays, in spite of high performance of users’ single channel behavior computing, it is still great challenge to understand users’ intention accurately from their multimodal behaviors. One reason for this challenge is that we still need to improve multimodal information fusion in theories, methodologies and practical systems. This paper presents a review of data fusion methods in multimodal human computer dialog. We first introduce the cognitive assumption of single channel processing, and then discuss its implementation methods in human computer dialog; for the task of multi-modal information fusion, serval computing models are presented after we introduce the principle description of multiple data fusion. Finally, some practical examples of multimodal information fusion methods are introduced and the possible and important breakthroughs of the data fusion methods in future multimodal human-computer interaction applications are discussed.

Speech communication integrated with other modalities

This brief paper is an editorial for the special issue on “Speech communication integrated with other modalities”. This special issue contains extended versions of selected topical papers from the 19th International Conference on Speech and Computer SPECOM-2017, organized on 12–16 September 2017 in Hatfield, UK. Five extended articles were selected for this special issue, all of which deal with speech-based human–computer communication jointly with visual, textual and/or other interaction modalities. The first three papers study various aspects of multimodal human–computer interaction, and in the remaining two papers, the authors study the video components of audio-visual speech recognition systems. In this editorial, we present an overview of the accepted articles and the selection process.

Measurement-Based Thermal Modeling Using Laser Thermography

There have been several studies in using infrared thermography for nondestructive evaluation and testing of materials. Some studies have explored the potential of using thermography for classification of materials based on their thermal response. However, the modeling behind these efforts is rather simplistic, and in some ways not corresponding well with practical applications. This paper presents a realistic model-based approach for characterizing unknown materials using laser thermography. The process being modeled is laser excitation step thermography with finite but not infinitesimal excitation length and uniform distribution over a finite radius. A mathematical model, based on the heat equation that governs the flow of heat in materials, is introduced to characterize objects based on their response to thermal excitation. Experimental results consistently demonstrated the ability of the proposed approach to classify different materials based on their thermal properties. This contactless characterization method might prove to be suitable for a range of applications including teleoperation, haptic mapping and multimodal human computer interaction.

Intelligence methods of multi-modal information fusion in human-computer interaction

We first introduce the concepts of single-modal information processing and multi-modal information fusion in cognitive science. Some classical multi-modal information fusion models and their computer implementations in history are also explained. Under the conditions that each channels information can be obtained, and their features could be unified representation synchronously, the fusion of multi-modal information can be transformed into classification or regression problems. For practical human-computer interaction systems, the performance of multi-modal information fusion largely relies on the accuracy of the single-modal information identification and the design of the interactive system. We present a practical example of multi-modal information fusion system, and discuss its performances on human computer interaction. Finally, the possible and important development trends for multi-modal human-computer interaction techniques and systems are discussed.

Fusion strategy-based multimodal human-computer interaction

Fusion strategy-based multimodal human-computer interaction