Multimodal Human-Computer Interaction System Research Articles

Multi-modal human-computer interaction system in cockpit

In order to explore new human-machine interaction methods, a set of multi-modal human-machine interaction coordinated control system is proposed, which realizes the basic flight control based on the change of the pilot’s field of view, touch control, voice control and other information obtained from multi-mode coordinated control. This system introduces a new type of human-computer interaction into the cockpit application on the basis of the human-computer interaction interface for flight driving, and has carried out research on multi-mode collaborative control system, mainly including eye movement interaction, touch interaction gesture Interaction and voice interaction. Finally, this project formed a multi-mode cooperative control software and hardware system.

Assistive Robotics: Adaptive Multimodal Interaction Improving People with Communication Disorders

Abstract: A significant portion of the Brazilian population consists of people with hearing or visual impairment. Multimodal interaction systems do not suit the needs of a mixed audience, since they are designed for a specific user profile. We propose a model for a Multimodal Human Computer Interaction System (MMHCI) based on services, embedded on an assistive robot, which is able to adapt the communication according to the type and degree of the user's disabilities. The proposed approach emphasizes adaption of interaction channels according to the user's needs. The experiments elicit positive user attitudes towards usability aspects of the system.

Adaptive deformable model for mouth boundary detection

University of SydneyDepartment of Electrical EngineeringSydney, New South Wales 2006AustraliaE-mail: armir@ee.usyd.edu.auKin-Man LamHong Kong Polytechnic UniversityDepartment of Electrical EngineeringHong KongAbstract. A new generalized algorithm is proposed to automatically ex-tract a mouth boundary model from human face images. Such an algo-rithm can contribute to human face recognition and lip-reading-assistedspeech recognition systems, in particular, and multimodal human com-puter interaction systems, in general. The new model is an iterative al-gorithm based on a hierarchical model adaptation scheme using deform-able templates, as a generalization of some of the previous works. Therole of prior knowledge is essential for perceptual organization in thealgorithm. The prior knowledge about the mouth shape is used to defineand initialize a primary deformable model. Each primary boundary curveof a mouth is formed on three control points, including two mouth cor-ners, whose locations are optimized using a primary energy functional.This energy functional essentially captures the knowledge of the mouthshape to perceptually organize image information. The primary model isfinely tuned in the second stage of optimization algorithm using a gener-alized secondary energy functional. Basically each boundary curve isfinely tuned using more control points. The primary model is replaced byan adapted model if there is an increase in the secondary energy func-tional. The results indicate that the new model adaptation technique sat-isfactorily generalizes the mouth boundary model extraction in an auto-mated fashion.