Multimodal Human Computer Research Articles

User expertise in speech centered multimodal human computer interaction

Multimodal interfaces aim to permit natural communication by speech and gesture. Typically the speech modality bears the principal information in the interaction with gesture complementing spoken commands. A continuing challenge is how to correlate and interpret the simultaneous inputs to estimate meaning and user intent. User expertise and familiarity figure prominently in the interpretation. The present research studies the effect of user expertise on multimodal human computer interaction. Users are classified into experienced and inexperienced depending on the amount of their exposure and interaction with multimodal systems. Each user is asked to perform simple tasks using a multimodal system. For each task the automatically recognized speech input is time stamped and the lag or lead of the gesture input is computed with respect to this time stamp. The time interval around the time stamp in which all the users’ gesture inputs occur is determined. For experienced users this interval averages 56.9% less than that for inexperienced users. The implication is that for experienced users the spoken input are the corresponding gesture input are more closely related in time than for inexperienced users. This behavior can be exploited in multimodal systems to increase efficiency and reduce time of response for the system.

Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals

We discuss the strong relationship between affect and cognition and the importance of emotions in multimodal human computer interaction (HCI) and user modeling. We introduce the overall paradigm for our multimodal system that aims at recognizing its users' emotions and at responding to them accordingly depending upon the current context or application. We then describe the design of the emotion elicitation experiment we conducted by collecting, via wearable computers, physiological signals from the autonomic nervous system (galvanic skin response, heart rate, temperature) and mapping them to certain emotions (sadness, anger, fear, surprise, frustration, and amusement). We show the results of three different supervised learning algorithms that categorize these collected signals in terms of emotions, and generalize their learning to recognize emotions from new collections of signals. We finally discuss possible broader impact and potential applications of emotion recognition for multimodal intelligent systems.

Speech and Language Processing for Multimodal Human-Computer Interaction

In this paper, we describe our recent work at Microsoft Research, in the project codenamed Dr. Who, aimed at the development of enabling technologies for speech-centric multimodal human-computer interaction. In particular, we present in detail MiPad as the first Dr. Who's application that addresses specifically the mobile user interaction scenario. MiPad is a wireless mobile PDA prototype that enables users to accomplish many common tasks using a multimodal spoken language interface and wireless-data technologies. It fully integrates continuous speech recognition and spoken language understanding, and provides a novel solution to the current prevailing problem of pecking with tiny styluses or typing on minuscule keyboards in today's PDAs or smart phones. Despite its current incomplete implementation, we have observed that speech and pen have the potential to significantly improve user experience in our user study reported in this paper. We describe in this system-oriented paper the main components of MiPad, with a focus on the robust speech processing and spoken language understanding aspects. The detailed MiPad components discussed include: distributed speech recognition considerations for the speech processing algorithm designs a stereo-based speech feature enhancement algorithm used for noise-robust front-end speech processings Aurora2 evaluation results for this front-end processings speech feature compression (source coding) and error protection (channel coding) for distributed speech recognition in MiPads HMM-based acoustic modeling for continuous speech recognition decodings a unified language model integrating context-free grammar and N-gram model for the speech decodings schema-based knowledge representation for the MiPad's personal information management tasks a unified statistical framework that integrates speech recognition, spoken language understanding and dialogue managements the robust natural language parser used in MiPad to process the speech recognizer's outputs a machine-aided grammar learning and development used for spoken language understanding for the MiPad tasks Tap & Talk multimodal interaction and user interface designs back channel communication and MiPad's error repair strategys and finally, user study results that demonstrate the superior throughput achieved by the Tap & Talk multimodal interaction over the existing pen-only PDA interface. These user study results highlight the crucial role played by speech in enhancing the overall user experience in MiPad-like human-computer interaction devices.

Using critical path analysis to model multimodal human–computer interaction

In this paper, the concept of multimodal human-computer interaction is explored. It is proposed that multimodality can be defined from human or technology perspectives, which place emphasis on different attributes of the system. Furthermore, in this paper it is argued that the most effective definition of multimodality concentrates on task and goal dependencies. Not only does this permit consideration over and above the technology/human distinction, but also allows consideration of multiple tasks. In order to explore this notion, critical path analysis is used to develop models of multimodal systems. The model describes multimodal HCI, and allows consideration of the effects of modality dependency. The models allow prediction of transaction time under various conditions. Predictions arising from these models are shown to be good fits with data obtained from user trials. Thus, it is proposed that one can develop and evaluate preliminary versions of multimodal systems prior to prototype development.

Analysing Cognitive Behaviour using LOTOS and Mexitl

Abstract. We argue that cognitive models should be used in analysing the usability of multi-modal human computer interfaces and further, that formal methods can be advantageously applied to such analysis. In pursuing this objective we specify the Interacting Cognitive Subsystems model formally using the process calculus LOTOS and then we verify that it satisfies certain behavioural goals formulated in the interval temporal logic Mexitl.

A space-time delay neural network for motion recognition and its application to lipreading.

Motion recognition has received increasing attention in recent years owing to heightened demand for computer vision in many domains, including the surveillance system, multimodal human computer interface, and traffic control system. Most conventional approaches classify the motion recognition task into partial feature extraction and time-domain recognition subtasks. However, the information of motion resides in the space-time domain instead of the time domain or space domain independently, implying that fusing the feature extraction and classification in the space and time domains into a single framework is preferred. Based on this notion, this work presents a novel Space-Time Delay Neural Network (STDNN) capable of handling the space-time dynamic information for motion recognition. The STDNN is unified structure, in which the low-level spatiotemporal feature extraction and high-level space-time-domain recognition are fused. The proposed network possesses the spatiotemporal shift-invariant recognition ability that is inherited from the time delay neural network (TDNN) and space displacement neural network (SDNN), where TDNN and SDNN are good at temporal and spatial shift-invariant recognition, respectively. In contrast to multilayer perceptron (MLP), TDNN, and SDNN, STDNN is constructed by vector-type nodes and matrix-type links such that the spatiotemporal information can be accurately represented in a neural network. Also evaluated herein is the performance of the proposed STDNN via two experiments. The moving Arabic numerals (MAN) experiment simulates the object's free movement in the space-time domain on image sequences. According to these results, STDNN possesses a good generalization ability with respect to the spatiotemporal shift-invariant recognition. In the lipreading experiment, STDNN recognizes the lip motions based on the inputs of real image sequences. This observation confirms that STDNN yields a better performance than the existing TDNN-based system, particularly in terms of the generalization ability. In addition to the lipreading application, the STDNN can be applied to other problems since no domain-dependent knowledge is used in the experiment.

Physicians in virtual environments — multimodal human–computer interaction

Modern tomography technologies like CT or MRI produce high-quality scans of the human anatomy. While conventional computer-aided image analysis falls back upon editing tomograms layer by layer, virtual environments offer enhanced visualization, image analysis and manipulation of the three-dimensional data sets. In this paper, the application of multimodal, user-oriented human–computer interaction is presented, facilitating and accelerating work with the tomographical data of individual patients. Hand gesture recognition is a major component of the interface, completed by speech understanding and further units like a 6-DOF mouse or acoustic feedback. Three-dimensional image segmentation, virtual bronchoscopy and virtual angioscopy are typical examples that illustrate the benefits of virtual environments for the realm of medicine.

Multimodal human-computer interaction

The European Research Consortium for Informatics and Mathematics (ERCIM) was established in April 1989 as a result of a joint initiative from CWI, GMD and INRIA. The Consortium is dedicated to the advancement of European research and development, in the areas of information technology and applied mathematics, and welcomes national research laboratories from all European countries as members. At present, ERCIM includes 11 national member institutions which are strongly involved in the European Community's R&D programmes: AEDIMA (Spain), CWI (the Netherlands), CNR (Italy), FORTH (Greece), GMD (Germany), INRIA (France), INESC (Portugal), RAL (United Kingdom), SICS (Sweden), SINTEF (Norway), VTT (Finland). Through the definition of common scientific goals and strategies, ERCIM member institutes aim to foster collaborative work within the European research community and to increase cooperation with European industry. To further these objectives, the Consortium organizes joint scientific Workshops and Advanced Courses, sponsors a Fellowship Programme for talented young researchers, undertakes joint strategic projects, encourages technology transfer, and promotes international research collaborations and networks. ERCIM publications include workshop proceedings, research and strategic reports, as well as a worldwide distributed quarterly newsletter. The Workshop on Multimodal Human-Computer Interaction that took place at INRIA-Lorraine in November 1993 is one of the numerous scientific events that ERCIM initiated and sponsored in 1993.