Chapter 12 - Multimodal Input

Abstract

This chapter reviews the advantages of multimodal interfaces and presents some examples of state-of-the-art multimodal systems. The focus is on the links between multimodality and cognition, namely the application of human cognitive processing models to improve understanding of multimodal behavior in different contexts, particularly in situations of high mental demand. The growing interest in multimodal interface design is inspired largely by the goal of supporting more flexible, powerfully expressive, and low cognitive load means of human–computer interaction (HCI). They have the potential to expand computing to more mobile and complex real-world applications, to be used by a broader spectrum of everyday people and to accommodate more adverse usage conditions than in the past. The state-of-the-art multimodal input systems are currently only able to process two to three modal input channels, are application-specific, and have limited interaction vocabularies and grammars. The two most mature types of multimodal systems, in which the keyboard and mouse have been completely replaced, are those that combine either speech and pen input or speech and lip movements.