Multimodal Communication, Natural Language and Direct Manipulation (Gestures) in Human-Computer Interaction

Abstract

Human-computer communication is quite impoverished compared to human-human communication, both in its form and its effectiveness. In many cases the aim of designing a multimedia interface is to produce an interaction which is more like that which takes place between people. Natural language is a highly developed form of (human-human) communication, one which is often supplemented by body language, one of the most important components of which is hand gesture. For example, people often use pointing (deixis) to avoid having to describe something that is complex, inexact or inaccessible to them. On the other hand, natural language can be much more refined at specifying what part or aspect of something is currently of interest. This is particularly true where quantification, abstraction, combination, negation or time are involved. This chapter describes how these two media of human-human interaction — natural language and gesture — can be integrated in a human-computer interface. It describes how these have been applied in a variety of different practical applications to combine the strengths of both modes of interaction.