Multimodal Human Computer Research Articles

Multimodal human computer interaction of wheelchairs supporting lower limb active rehabilitation

Currently, an important challenge in stroke rehabilitation is how to effectively restore motor functions of lower limbs. This paper presents multimodal human computer interaction (HCI) of wheelchairs supporting lower limb active rehabilitation. First, multimodal HCI incorporating motor imagery electroencephalography (EEG), electromyography (EMG) and speech is designed. Second, prototype supporting wheelchair active rehabilitation method is illustrated in details. Third, the preliminary brain-computer interfaces (BCI) and speech recognition task experiments are carried out respectively, and the results are obtained. Finally, discussion is conducted and conclusion is drawn. This study has important practical significance in auxiliary movements and neurorehabilitation for stroke patients.

Speech recognition and intelligent translation under multimodal human–computer interaction system

Abstract The traditional translation robot is limited to the translation of single-mode text images and text videos, which has the problem of low translation accuracy. Therefore, speech recognition and intelligent translation in multimodal human–computer interaction (HCI) system are proposed. First, the network structure of speech recognition model in multi-channel HCI system is established, and the multi-head self-attention mechanism is constructed. Then, the artificial intelligence voice wake-up function is designed, and a multimodal machine translation model is constructed. On this basis, selective attention is added to obtain visual recognition of perceived text, and the decoder is used for multimodal gating fusion to realize the output of encoder translation results. Experimental results show that this method has high BLUE value and high translation accuracy.

Research on Intelligent Wheelchair Multimode Human–Computer Interaction and Assisted Driving Technology

The traditional wheelchair focuses on the “human-chair” motor function interaction to ensure the elderly and people with disabilities’ basic travel. For people with visual, hearing, physical disabilities, etc., the current wheelchairs show shortcomings in terms of accessibility and independent travel for this group. Therefore, this paper develops an intelligent wheelchair with multimodal human–computer interaction and autonomous navigation technology. Firstly, it researches the multimodal human–computer interaction technology of occupant gesture recognition, speech recognition, and head posture recognition and proposes a wheelchair control method of three-dimensional head posture mapping the two-dimensional plane. After testing, the average accuracy of the gesture, head posture and voice control modes of the motorized wheelchair proposed in this study reaches more than 95 percent. Secondly, the LiDAR-based smart wheelchair indoor autonomous navigation technology is investigated to realize the autonomous navigation of the wheelchair by constructing an environment map, using A* and DWA algorithms for global and local path planning, and adaptive Monte Carlo simulation algorithms for real-time localization. Experiments show that the position error of the wheelchair is within 10 cm, and the heading angle error is less than 5° during the autonomous navigation. The multimode human–computer interaction and assisted driving technology proposed in this study can partially compensate and replace the functional deficiencies of the disabled population and improve the quality of life of the elderly and disabled population.

Modelling conversational agent with empathy mechanism

Empathy mechanism in communication is the cornerstone for effective and meaningful interaction. Establishing an empathy mechanism in conversation agent (CA) requires accurate recognition of users’ emotions to facilitate generates appropriate empathetic responses. Therefore, we proposed a Multimodal Emotion Recognition Model (MERM) to recognizes a user’s emotional state from multimodal data (audio, facial expressions, and conversation text) during conversation, and an Interactive Empathetic Conversation Model (IECM) to generate empathetic responses based on the MERM. Comparative and ablation study results indicated that the proposed models outperform existing methods in recognizing the user’s emotions and generating appropriate empathetic responses. We also conducted an experiment study, the results indicated that the CA significantly enhances the user’s emotional experience.

Perception and Decision-Making for Multi‑Modal Interaction Based on Fuzzy Theory in the Dynamic Environment

The projection-augmented reality assembly assistance system, which supports natural multi-modal human–computer interaction, has the potential to improve interaction efficiency. However, the effectiveness of the interaction method is compromised by the interference caused by the dynamically changing industrial environment. To address this issue, this study proposed a multi-model interaction model based on fuzzy theory and developed an environment perception decision-making interaction system. This system continuously monitors real-time environment data, evaluates interaction performance, and visually recommends the optimal method to assist users in completing their tasks. A comparative experiment was conducted between the proposed system and a non-environment perception interaction system. The collected data, including interaction success rate, task time, system usability, and user experience, were analyzed. The results demonstrated that the proposed system effectively enhances the adaptability and comfort of computer systems in complex environments and improves communication efficiency between humans and machines.

A MULTI-MODAL INTENTION INTEGRATION METHOD FOR UNMANNED AERIAL VEHICLES BASED ON THE COMBINATION OF BIOELECTRICAL SIGNALS AND CONVENTIONAL HUMAN–COMPUTER INTERACTION

With the rapid development of intelligent information technology, human–computer interaction for unmanned aerial vehicle (UAV) systems has gradually developed from single-mode interaction to multi-modal interaction. Due to the problems of information ambiguity and operational redundancy in multi-modal interaction, and the inability to effectively realize the coupling of human–computer interaction of different modalities, a priority strategy for multi-modal interaction for UAV systems is proposed. Living cells or tissues, whether in a static or active state, will produce regular electrical phenomena called bioelectricity. Human–computer interaction intention recognition technology based on bioelectrical signals can significantly enhance the real-time performance and flexibility of human–computer interaction and has great development potential. The characteristics of human–computer interaction intention recognition methods based on bioelectrical signals, such as electroencephalograms (EEGs), electromyograms (EMGs), electrooculograms (EOGs), and other conventional methods are introduced and analyzed, and then the basic characteristics of single-mode interaction and multi-modal interaction are compared. Furthermore, the priority strategies for UAV systems under different interaction modes are studied based on the analysis of task value evaluation parameters. The cumulative value, urgency, and task complexity are comprehensively analyzed and evaluated, and a set of priority strategies for multi-modal interaction is integrated. Through the simulation and verification of the multi-modal human–computer interaction software of the UAV system, it is determined that the proposed priority strategy conforms to the operation intention of the user, which improves the interaction efficiency and the reliability. In noisy environments, the UAV system control can be combined with the complementary information of various interaction methods to achieve precise control, which has obvious advantages.

Magnetic Motion Tracking for Natural Human Computer Interaction: A Review

In the field of multimodal human–computer interaction (HCI), hand (and finger) motion tracking remains a critical challenge because it is the input means of visual reproduction and provides the necessary parameters for real-time modeling of the haptic model. Among motion tracking methods including optical tracking methods, inertial tracking methods, and magnetic motion tracking (MMT) methods, the MMT method has a great development prospect as a marker-based method because of its advantages of no line-of-sight problem, being natural-oriented, and high accuracy. In this article, we first discuss the performance and application scenarios under different configurations of electromagnet-based MMT (EMMT) and permanent-magnet-based MMT (PMMT) systems. And then, the MMT algorithms are reviewed, which can be divided into two broad categories: The model-based algorithms include linear algorithm, nonlinear optimization algorithm, and recursive Bayesian algorithm, and the offline-data-based algorithms include neural network algorithm and lookup table (LUT) algorithm. In addition, a state-of-the-art comparison of MMT algorithms is given. In the end, we have an insightful discussion and give the expected future outlook of MMT.

Multimodal Natural Human–Computer Interfaces for Computer-Aided Design: A Review Paper

Computer-aided design (CAD) systems have advanced to become a critical tool in product design. Nevertheless, they still primarily rely on the traditional mouse and keyboard interface. This limits the naturalness and intuitiveness of the 3D modeling process. Recently, a multimodal human–computer interface (HCI) has been proposed as the next-generation interaction paradigm. Widening the use of a multimodal HCI provides new opportunities for realizing natural interactions in 3D modeling. In this study, we conducted a literature review of a multimodal HCI for CAD to summarize the state-of-the-art research and establish a solid foundation for future research. We explore and categorize the requirements for natural HCIs and discuss paradigms for their implementation in CAD. Following this, factors to evaluate the system performance and user experience of a natural HCI are summarized and analyzed. We conclude by discussing challenges and key research directions for a natural HCI in product design to inspire future studies.

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Multimodal human–computer interaction (HCI) systems pledge a more human–human-like interaction between machines and humans. Their prowess in emanating an unambiguous information exchange between the two makes these systems more reliable, efficient, less error prone, and capable of solving complex tasks. Emotion recognition is a realm of HCI that follows multimodality to achieve accurate and natural results. The prodigious use of affective identification in e-learning, marketing, security, health sciences, etc., has increased demand for high-precision emotion recognition systems. Machine learning (ML) is getting its feet wet to ameliorate the process by tweaking the architectures or wielding high-quality databases (DB). This paper presents a survey of such DBs that are being used to develop multimodal emotion recognition (MER) systems. The survey illustrates the DBs that contain multi-channel data, such as facial expressions, speech, physiological signals, body movements, gestures, and lexical features. Few unimodal DBs are also discussed that work in conjunction with other DBs for affect recognition. Further, VIRI, a new DB of visible and infrared (IR) images of subjects expressing five emotions in an uncontrolled, real-world environment, is presented. A rationale for the superiority of the presented corpus over the existing ones is instituted.

Development of an Integrated Haptic Sensor System for Multimodal Human–Computer Interaction Using Ultrasonic Array and Cable Robot

This paper presents the design and development of a novel haptic sensor system. It integrates cable-driven force and ultrasonic tactile feedback, which can produce multimodal haptic stimuli. The sensing element includes a Leapmotion, an ultrasonic transducer array, tension sensors, and rotary encoders, which are used to capture hand posture, project tactile points, measure cable force and length, respectively. Firstly, a 6-DOF cable-driven force feedback apparatus based on parallel mechanism is designed and ultrasonic phased array is combined to form a multimodal haptic feedback system. Secondly, a multimodal haptic fusion method for cable-driven force and ultrasonic tactile is firstly proposed to invoke realistic compound haptic sensations. To enhance the rendering effects of each subsystem, admittance control is developed for a cable robot, and a new perceived magnitude model is established for ultrasound tactile rendering. A psychophysical experiment is conducted to study the perceived characteristics of multimodal haptic stimuli. To verify the proposed system, a series of experiments were carried out, whose results indicated that the system performs well at multi-property haptic rendering and confirm the accuracy and sensitive advantage of our system in virtual reality applications. The results of our study indicate that this device has great application potential in human-computer interaction.

Systematic Review of Multimodal Human–Computer Interaction

This document presents a systematic review of Multimodal Human–Computer Interaction. It shows how different types of interaction technologies (virtual reality (VR) and augmented reality, force and vibration feedback devices (haptics), and tracking) are used in different domains (concepts, medicine, physics, human factors/user experience design, transportation, cultural heritage, and industry). A systematic literature search was conducted identifying 406 articles initially. From these articles, we selected 112 research works that we consider most relevant for the content of this article. The articles were analyzed in-depth from the viewpoint of temporal patterns, frequency of usage in types of technology in different domains, and cluster analysis. The analysis allowed us to answer relevant questions in searching for the next steps in work related to multimodal HCI. We looked at the typical technology type, how the technology type and frequency have changed in time over each domain, and how papers are grouped across metrics given their similarities. This analysis determined that VR and haptics are the most widely used in all domains. While VR is the most used, haptic interaction is presented in an increasing number of applications, suggesting future work on applications that configure VR and haptic together.

Utilisation of Embodied Agents in the Design of Smart Human–Computer Interfaces—A Case Study in Cyberspace Event Visualisation Control

The goal of the research reported here was to investigate whether the design methodology utilising embodied agents can be applied to produce a multi-modal human–computer interface for cyberspace events visualisation control. This methodology requires that the designed system structure be defined in terms of cooperating agents having well-defined internal components exhibiting specified behaviours. System activities are defined in terms of finite state machines and behaviours parameterised by transition functions. In the investigated case the multi-modal interface is a component of the Operational Centre which is a part of the National Cybersecurity Platform. Embodied agents have been successfully used in the design of robotic systems. However robots operate in physical environments, while cyberspace events visualisation involves cyberspace, thus the applied design methodology required a different definition of the environment. It had to encompass the physical environment in which the operator acts and the computer screen where the results of those actions are presented. Smart human–computer interaction (HCI) is a time-aware, dynamic process in which two parties communicate via different modalities, e.g., voice, gesture, eye movement. The use of computer vision and machine intelligence techniques are essential when the human is carrying an exhausting and concentration demanding activity. The main role of this interface is to support security analysts and operators controlling visualisation of cyberspace events like incidents or cyber attacks especially when manipulating graphical information. Visualisation control modalities include visual gesture- and voice-based commands.

INSMA: An integrated system for multimodal data acquisition and analysis in the intensive care unit.

Modern intensive care units (ICU) are equipped with a variety of different medical devices to monitor the physiological status of patients. These devices can generate large amounts of multimodal data daily that include physiological waveform signals (arterial blood pressure, electrocardiogram, respiration), patient alarm messages, numeric vitals data, etc. In order to provide opportunities for increasingly improved patient care, it is necessary to develop an effective data acquisition and analysis system that can assist clinicians and provide decision support at the patient bedside. Previous research has discussed various data collection methods, but a comprehensive solution for bedside data acquisition to analysis has not been achieved. In this paper, we proposed a multimodal data acquisition and analysis system called INSMA, with the ability to acquire, store, process, and visualize multiple types of data from the Philips IntelliVue patient monitor. We also discuss how the acquired data can be used for patient state tracking. INSMA is being tested in the ICU at University Hospitals Cleveland Medical Center.

Framing the Design Space of Multimodal Mid-Air Gesture and Speech-Based Interaction With Mobile Devices for Older People

Multimodal human–computer interaction has been sought to provide not only more compelling interactive experiences, but also more accessible interfaces to mobile devices. With the advance in mobile technology and in affordable sensors, multimodal research that leverages and combines multiple interaction modalities (such as speech, touch, vision, and gesture) has become more and more prominent. This article provides a framework for the key aspects in mid-air gesture and speech-based interaction for older adults. It explores the literature on multimodal interaction and older adults as technology users and summarises the main findings for this type of users. Building on these findings, a number of crucial factors to take into consideration when designing multimodal mobile technology for older adults are described. The aim of this work is to promote the usefulness and potential of multimodal technologies based on mid-air gestures and voice input for making older adults' interaction with mobile devices more accessible and inclusive.

A novel gaze-supported multimodal human-computer interaction for ultrasound machines.

Conventional ultrasound (US) machines employ a physical control panel (PCP) as the primary user interface for machine control. This panel is adjacent to the main machine display that requires the operator's constant attention. The switch of attention to the control panel can lead to interruptions in the flow of the medical examination. Some ultraportable machines also lack many physical controls. Furthermore, the need to both control the US machine and observe the US image may lead the practitioners to adopt unergonomic postures and repetitive motions that can lead to work-related injuries. Therefore, there is a need for a more efficient human-computer interaction method on US machines. To tackle some of the limitations with the PCP, we propose to merge the PCP into the main screen of the US machines. We propose to use gaze tracking and a handheld controller so that machine control can be achieved via a multimodal human-computer interaction (HCI) method that does not require one to touch the screen or look away from the US image. As a first step, a pop-up menu and measurement tool were designed on top of the US image based on gaze position for efficient machine control. A comparative study was performed on the BK Medical SonixTOUCH US machine. Participants were asked to complete the task of measuring the area of an ellipse-shaped tumor in a phantom using our gaze-supported HCI method as well as the traditional method. The user study indicates that the task completion time can be reduced by [Formula: see text] when using our gaze-supported HCI, while no extra workload is imposed on the operators. Our preliminary study suggests that, when combined with a simple handheld controller, eye gaze tracking can be integrated into the US machine HCI for more efficient machine control.

Data fusion methods in multimodal human computer dialog

In multimodal human computer dialog, non-verbal channels, such as facial expression, posture, gesture, etc, combined with spoken information, are also important in the procedure of dialogue. Nowadays, in spite of high performance of users’ single channel behavior computing, it is still great challenge to understand users’ intention accurately from their multimodal behaviors. One reason for this challenge is that we still need to improve multimodal information fusion in theories, methodologies and practical systems. This paper presents a review of data fusion methods in multimodal human computer dialog. We first introduce the cognitive assumption of single channel processing, and then discuss its implementation methods in human computer dialog; for the task of multi-modal information fusion, serval computing models are presented after we introduce the principle description of multiple data fusion. Finally, some practical examples of multimodal information fusion methods are introduced and the possible and important breakthroughs of the data fusion methods in future multimodal human-computer interaction applications are discussed.

Speech communication integrated with other modalities

This brief paper is an editorial for the special issue on “Speech communication integrated with other modalities”. This special issue contains extended versions of selected topical papers from the 19th International Conference on Speech and Computer SPECOM-2017, organized on 12–16 September 2017 in Hatfield, UK. Five extended articles were selected for this special issue, all of which deal with speech-based human–computer communication jointly with visual, textual and/or other interaction modalities. The first three papers study various aspects of multimodal human–computer interaction, and in the remaining two papers, the authors study the video components of audio-visual speech recognition systems. In this editorial, we present an overview of the accepted articles and the selection process.

Measurement-Based Thermal Modeling Using Laser Thermography

There have been several studies in using infrared thermography for nondestructive evaluation and testing of materials. Some studies have explored the potential of using thermography for classification of materials based on their thermal response. However, the modeling behind these efforts is rather simplistic, and in some ways not corresponding well with practical applications. This paper presents a realistic model-based approach for characterizing unknown materials using laser thermography. The process being modeled is laser excitation step thermography with finite but not infinitesimal excitation length and uniform distribution over a finite radius. A mathematical model, based on the heat equation that governs the flow of heat in materials, is introduced to characterize objects based on their response to thermal excitation. Experimental results consistently demonstrated the ability of the proposed approach to classify different materials based on their thermal properties. This contactless characterization method might prove to be suitable for a range of applications including teleoperation, haptic mapping and multimodal human computer interaction.

MMQuiz: multi modal human computer interaction-based online assessment system

Ever changing computing techniques and novel innovations in the education system plays a major role for the way any institutions prefer to adopt now-a-days. All the varsities and institutions started transferring their mode of education from traditional class room-based to internet-based online. Part of these processes, the way how students can be assessed online also plays a major role. In this work, a framework has been proposed which was implemented so as to enable the physically challenged students to efficiently utilise to complete their online-based assessments. Though many voice-based browsers existed, they were all far from reality. Several frameworks were designed for specified objectives like interactive assessment whilst they supported generic features like browsing and mailing. Any individual who may be able to communicate via speech may utilise our proposed framework. The same framework could be employed in any varsities or institutions to conduct their speech-based assessments online.

MMQuiz: multi modal human computer interaction-based online assessment system

MMQuiz: multi modal human computer interaction-based online assessment system